A bi-modal face recognition framework integrating facial expression with facial appearance

This chapter assessed the possibility of using facial behaviour as another individual trait for personal identification and identification improvement. Facial expression variations were previously thought of as noise that would degrade the classification performance, so

Tuning functions for automatic detection of brief changes of facial expression in the human brain

Over the course of the last decade, infrared (IR) and particularly thermal IR imaging based face recognition has emerged as a promising complement to conventional, visible spectrum based approaches which continue to struggle when applied in the real world. While inherently insensitive to visible spectrum illumination changes, IR images introduce specific challenges of their own, most notably sensitivity to factors which affect facial heat emission patterns, e.g. emotional state, ambient temperature, and alcohol intake. In addition, facial expression and pose changes are more difficult to correct in IR images because they are less rich in high frequency detail which is an important cue for fitting any deformable model. In this paper we describe a novel method which addresses these major challenges. Specifically, to normalize for pose and facial expression changes we generate a synthetic frontal image of a face in a canonical, neutral facial expression from an image of the face in an arbitrary pose and facial expression. This is achieved by piecewise affine warping which follows active appearance model (AAM) fitting. This is the first publication which explores the use of an AAM on thermal IR images; we propose a pre-processing step which enhances detail in thermal images, making AAM convergence faster and more accurate. To overcome the problem of thermal IR image sensitivity to the exact pattern of facial temperature emissions we describe a representation based on reliable anatomical features. In contrast to previous approaches, our representation is not binary; rather, our method accounts for the reliability of the extracted features. This makes the proposed representation much more robust both to pose and scale changes. The effectiveness of the proposed approach is demonstrated on the largest public database of thermal IR images of faces on which it achieved 100% identification rate, significantly outperforming previously described methods.

BackgroundNeurological disorders, ranging from common conditions like Alzheimer’s disease that is a progressive neurodegenerative disorder and remains the most common cause of dementia worldwide to rare disorders such as Angelman syndrome, impose a significant global health burden. Altered facial expressions are a common symptom across these disorders, potentially serving as a diagnostic indicator. Deep learning algorithms, especially convolutional neural networks (CNNs), have shown promise in detecting these facial expression changes, aiding in diagnosing and monitoring neurological conditions.ObjectivesThis systematic review and meta-analysis aimed to evaluate the performance of deep learning algorithms in detecting facial expression changes for diagnosing neurological disorders.MethodsFollowing PRISMA2020 guidelines, we systematically searched PubMed, Scopus, and Web of Science for studies published up to August 2024. Data from 28 studies were extracted, and the quality was assessed using the JBI checklist. A meta-analysis was performed to calculate pooled accuracy estimates. Subgroup analyses were conducted based on neurological disorders, and heterogeneity was evaluated using the I2 statistic.ResultsThe meta-analysis included 24 studies from 2019 to 2024, with neurological conditions such as dementia, Bell’s palsy, ALS, and Parkinson’s disease assessed. The overall pooled accuracy was 89.25% (95% CI 88.75–89.73%). High accuracy was found for dementia (99%) and Bell’s palsy (93.7%), while conditions such as ALS and stroke had lower accuracy (73.2%).ConclusionsDeep learning models, particularly CNNs, show strong potential in detecting facial expression changes for neurological disorders. However, further work is needed to standardize data sets and improve model robustness for motor-related conditions.

Whether the brain represents facial expressions as perceptual continua or as emotion categories remains controversial. Here, we measured the neural response to morphed images to directly address how facial expressions of emotion are represented in the brain. We found that face-selective regions in the posterior superior temporal sulcus and the amygdala responded selectively to changes in facial expression, independent of changes in identity. We then asked whether the responses in these regions reflected categorical or continuous neural representations of facial expression. Participants viewed images from continua generated by morphing between faces posing different expressions such that the expression could be the same, could involve a physical change but convey the same emotion, or could differ by the same physical amount but be perceived as two different emotions. We found that the posterior superior temporal sulcus was equally sensitive to all changes in facial expression, consistent with a continuous representation. In contrast, the amygdala was only sensitive to changes in expression that altered the perceived emotion, demonstrating a more categorical representation. These results offer a resolution to the controversy about how facial expression is processed in the brain by showing that both continuous and categorical representations underlie our ability to extract this important social cue.

This paper presents a novel Gabor-based kernel Principal Component Analysis (PCA) method by integrating the Gabor wavelet representation of face images and the kernel PCA method for face recognition. Gabor wavelets first derive desirable facial features characterized by spatial frequency, spatial locality, and orientation selectivity to cope with the variations due to illumination and facial expression changes. The kernel PCA method is then extended to include fractional power polynomial models for enhanced face recognition performance. A fractional power polynomial, however, does not necessarily define a kernel function, as it might not define a positive semidefinite Gram matrix. Note that the sigmoid kernels, one of the three classes of widely used kernel functions (polynomial kernels, Gaussian kernels, and sigmoid kernels), do not actually define a positive semidefinite Gram matrix either. Nevertheless, the sigmoid kernels have been successfully used in practice, such as in building support vector machines. In order to derive real kernel PCA features, we apply only those kernel PCA eigenvectors that are associated with positive eigenvalues. The feasibility of the Gabor-based kernel PCA method with fractional power polynomial models has been successfully tested on both frontal and pose-angled face recognition, using two data sets from the FERET database and the CMU PIE database, respectively. The FERET data set contains 600 frontal face images of 200 subjects, while the PIE data set consists of 680 images across five poses (left and right profiles, left and right half profiles, and frontal view) with two different facial expressions (neutral and smiling) of 68 subjects. The effectiveness of the Gabor-based kernel PCA method with fractional power polynomial models is shown in terms of both absolute performance indices and comparative performance against the PCA method, the kernel PCA method with polynomial kernels, the kernel PCA method with fractional power polynomial models, the Gabor wavelet-based PCA method, and the Gabor wavelet-based kernel PCA method with polynomial kernels.

Problem statement: Face occlusion is a very challenging problem in fac e recognition. The performance of face recognition system decreases dr astically due to the presence of partial occlusion on the face. Extracting discriminative features to achieve accurate detection versus computational overhead in extracting the features, which affects the classification speed, would be a sustained problem. The objective of this study is to segment the human face into non-occluded and occluded part of the occluded human face image. In General, for f ace detection special facial features are extracted . In the proposed study a simplified algorithm to ext ract the features is developed. Approach: An algorithm which enables the automatic detection of the presence of occlusions on the face would be a useful tool to increase the performances of the sys tem. The face image was preprocessed to enhance the input face images in order to reduce the loss o f classification performance due to changes in faci al appearance. The experiment also balances both illumination and facial expression changes. Results: In this study, a Mean Based Weight Matrix (MBWM) algorithm has been proposed to enhance the performance by 4.25% than the LBP method. Conclusion: The proposed model has been tested on occluded face images with a dataset obtained from t he MIT face database.

Objective. Interferences from spatially adjacent non-target stimuli are known to evoke event-related potentials (ERPs) during non-target flashes and, therefore, lead to false positives. This phenomenon was commonly seen in visual attention-based brain–computer interfaces (BCIs) using conspicuous stimuli and is known to adversely affect the performance of BCI systems. Although users try to focus on the target stimulus, they cannot help but be affected by conspicuous changes of the stimuli (such as flashes or presenting images) which were adjacent to the target stimulus. Furthermore, subjects have reported that conspicuous stimuli made them tired and annoyed. In view of this, the aim of this study was to reduce adjacent interference, annoyance and fatigue using a new stimulus presentation pattern based upon facial expression changes. Our goal was not to design a new pattern which could evoke larger ERPs than the face pattern, but to design a new pattern which could reduce adjacent interference, annoyance and fatigue, and evoke ERPs as good as those observed during the face pattern. Approach. Positive facial expressions could be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast is big enough to evoke strong ERPs. In this paper, a facial expression change pattern between positive and negative facial expressions was used to attempt to minimize interference effects. This was compared against two different conditions, a shuffled pattern containing the same shapes and colours as the facial expression change pattern, but without the semantic content associated with a change in expression, and a face versus no face pattern. Comparisons were made in terms of classification accuracy and information transfer rate as well as user supplied subjective measures. Main results. The results showed that interferences from adjacent stimuli, annoyance and the fatigue experienced by the subjects could be reduced significantly (p < 0.05) by using the facial expression change patterns in comparison with the face pattern. The offline results show that the classification accuracy of the facial expression change pattern was significantly better than that of the shuffled pattern (p < 0.05) and the face pattern (p < 0.05). Significance. The facial expression change pattern presented in this paper reduced interference from adjacent stimuli and decreased the fatigue and annoyance experienced by BCI users significantly (p < 0.05) compared to the face pattern.

Student engagement during online instruction has been a challenge. Particularly in an asynchronous learning environment, assessing and measuring student engagement is a difficult task. This paper proposes a solution to fill in the gap between the need to evaluate student engagement during online instruction and the lack of an effective method to do so. The proposed web-based learning environment fully incorporates the latest development in facial expression recognition to capture the facial expression changes during the course of instruction. The facial expression data can then be visualized for further analysis to help instructors pinpoint the exact locations where the instruction needs improvements. In addition, the proposed system can also be used in corporate training settings, particularly in scenarios where the learners' emotional responses are desired.

In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailored for different individuals. It takes the animation parameters as input and exploits an attention mask to manipulate facial expression changes for the input individuals. Furthermore, to better establish authentic correspondences between visual motions (i.e., facial expression changes and head movements) and audios, we leverage a high-accuracy motion capture dataset instead of relying on long videos of specific individuals. After attaining the visual and audio correspondences, we can effectively train our network in an end-to-end fashion. Extensive experiments on qualitative and quantitative results demonstrate that our algorithm achieves high-quality photo-realistic talking-head videos including various facial expressions and head motions according to speech rhythms and outperforms the state-of-the-art.

In this paper, we propose a novel face recognition method based on local appearance feature extraction using dual-tree complex wavelet transform (DT-CWT). It provides a local multiscale description of images with good directional selectivity and invariance to shifts and in-plane rotations. In the dual-tree implementation, two parallel discrete wavelet transform (DWT) with different lowpass and highpass filters in different scales were used. The linear combination of subbands generated by two parallel DWT is used to generate 6 different directional subbands with complex coefficients. It is insensitive to illumination variations and facial expression changes. 2-D dual-tree complex wavelet transform is less redundant and computationally efficient. The local DT-CWT coefficients are used to extract the facial features which improve the face recognition with small sample size in less computation. The local features based methods have been successfully applied to face recognition and achieved state-of-the-art performance. Normally most of the local appearance based methods the facial features are extracted from several local regions and concatenated into an enhanced feature vector as a face descriptor. In this approach we divide the face into several (m ? m) non-overlapped parallelogram blocks instead of square or rectangle blocks. The local mean, standard deviation and energy of complex wavelet coefficients are used to describe the face image. Experiments, on two well-known databases, namely, Yale and ORL databases, shows the Local DT-CWT approach performs well on illumination, expression and perspective variant faces with single sample compared to PCA and global DT-CWT. Furthermore, in addition to the consistent and promising classification performances, our proposed Local DT-CWT based method has a really low computational complexity.

Local fusion of complex dual-tree wavelet coefficients based face recognition for single sample problem

Interferences from the spatially adjacent non-target stimuli evoke ERPs during non-target sub-trials and lead to false positives. This phenomenon is commonly seen in visual attention based BCIs and affects the performance of BCI system. Although, users or subjects tried to focus on the target stimulus, they still could not help being affected by conspicuous changes of the stimuli (flashes or presenting images) which were adjacent to the target stimulus. In view of this case, the aim of this study is to reduce the adjacent interference using new stimulus presentation pattern based on facial expression changes. Positive facial expressions can be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast will be big enough to evoke strong ERPs. In this paper, two different conditions (Pattern_1, Pattern_2) were used to compare across objective measures such as classification accuracy and information transfer rate as well as subjective measures. Pattern_1 was a “flash-only” pattern and Pattern_2 was a facial expression change of a dummy face. In the facial expression change patterns, the background is a positive facial expression and the stimulus is a negative facial expression. The results showed that the interferences from adjacent stimuli could be reduced significantly (P<0.05) by using the facial expression change patterns. The online performance of the BCI system using the facial expression change patterns was significantly better than that using the “flash-only” patterns in terms of classification accuracy (p<0.01), bit rate (p<0.01), and practical bit rate (p<0.01). Subjects reported that the annoyance and fatigue could be significantly decreased (p<0.05) using the new stimulus presentation pattern presented in this paper.

This paper deals with the problem of 3D alignment of faces in the presence of some facial expression changes. The data are three dimensional and obtained using a laser scanner. Our approach is based on the differential geometry of the surface and computes the intrinsic local fiducial points on the surface and on curves that reside on the surface. Because these fiducial points are local, they allow partial alignment, where part of the face is viewed. Moreover, since the fiducial points are relatively affine‐invariant to local affine transformations, they allow matching and alignment when facial expression changes affect part of the face. A fast, noniterative alignment procedure is presented in this paper that establishes reliable correspondences between fiducial points without any prior knowledge of the overall nonlinear global transformation that takes place after the changes in facial expressions. This is achieved through the construction of a set of ordered novel absolute local affine invariants. With enough fiducial points set as correspondents, the overall nonlinear transformation is computed and the face before and after the transformation are aligned. For comparison, we also compare the alignment performance of our method with that of the iterative closest point (ICP) method and the coherent point drift (CPD) method which is based on the Gaussian mixture model by looking at the between‐to‐within variation (signal‐to‐noise ratio, SNR) between these two classes (match vs nonmatch) based on the average alignment errors for the genuine matches and nonmatches, normalized by their respective within variation, and by running it on the 3D face database GavabDB. The separation between the true match and nonmatch is best for our zero‐torsion method (SNR of 1.2), followed by the ICP method (SNR of 0.37)and the CPD method (SNR of 0.15). It is interesting to observe here that, although the alignment error for the true match cases is lowest for the CPD method, it is also extremely low for the nonmatch cases, which would mean that the method would force a query into a wrong face, yielding poor specificity (lots of false alarms). © 2013 Institute of Electrical Engineers of Japan. Published by John Wiley &amp; Sons, Inc.

Facial expression recognition is an effective method for identifying someone's emotional expression. Emotional expressions can be recognized from changes in facial expressions, wrinkles on the forehead, blinking of the eyes, or changes in facial skin color. Facial expressions that a person generally has, such as neutral, angry, happy expressions. The problem that often occurs is the subjective assessment of a person's expression. This research examines how artificial intelligence can recognize facial expressions. The facial recognition process in the research uses a Convolutional Neural Network (CNN), which is a deep learning method capable of carrying out an independent learning process for object recognition, object extraction and classification and can be applied to high resolution images that have a nonparametric distribution model. The two main stages in CNN are feature learning and classification. The results of facial expression recognition can be used to detect a person's emotions. This research uses the FER2013 dataset which contains images of happy, sad, angry, afraid, surprised, disgusted and neutral emotions. The data set in the research received tests that had been carried out, namely the percentage of accuracy level in the model was 76%. It is hoped that the classification of emotions resulting from this research can contribute to the development of artificial intelligence technology and as a tool in various fields such as psychology, education and others. For further research, it can be developed further by adding other architectures such as VGG19, MobileNet, and ResNet-50 so that the resulting CNN model is more optimal.