The Fuzzy Emotion Recognition Framework Using Semantic-Linguistic Facial Features

The human face has a great accumulation and a diversity of facial expressions. It explores the feelings of a person and can be used to judge the emotional intents of the person to a certain level. By using facial detection and recognition systems, varieties of applications are working in computer vision, surveillance system, security, authentication, or verification of a person and home automation system based on digital image processing with the help of the Internet of Things. The state of the art in these applications is to detect expressions with their intensity level. It is an attention-grabbing problem due to the complex nature of facial features, which is associated with emotions. For that purpose, it is essential to develop an innovative deep learning model to detect and estimate the facial expression intensity level. To do this, a multi-level deep convolutional neural network is proposed to recognize facial expression and their intensity level. At the first level, Expression-Net classifies face expressions, and at the second level, Intensity-Net estimates the intensity of the facial expression. Evaluation of the proposed model for facial expression recognition and intensity estimation is carried out by using the extended Cohn–Kanade and Japanese Female Facial Expression datasets. The proposed method shows an outstanding performance in terms of accuracy of 98.8% and 97.7% for both the datasets as compared to state-of-the-art techniques.

Facial expression analysis is one of the most essential tools for behavior interpretation and emotion modeling in Intelligent Human-Computer Interaction (HCI). Although humans can easily interpret facial emotions, computers have great difficulty doing so. Analyzing changes and deformations in the face is one of the methods through which machines can interpret facial expressions. However, maintaining great precision while being accurate, stable, and quick is still challenging in this field. To address this issue, this research presents an innovative and novel method to fully automatically extract critical features from a face during a facial expression. Various machine learning models are used on these features to analyze emotions. We used the optical flow algorithm to extract motion vectors divided into sections on the subject’s face. Finally, each section and its symmetric section were used to calculate a new vector. The final features produce a state-of-the-art accuracy of over 98% in emotion classification in the Extended Cohen-Kanade (CK+) facial expression dataset. Furthermore, we proposed an algorithm to filter the most important features with an SVM classifier and achieved an accuracy of over 97 % by only looking at 15% of the face area.

Face recognition is a biometric technology used to identify individuals based on facial feature information. Previous studies of face recognition used features including the eye, mouth and nose; however, there have been few studies on the effects of using other facial components, such as the eyebrows and chin, on recognition performance. We measured the recognition accuracy affected by these facial components, and compared the differences between computer-based and human-based facial recognition methods. This research is novel in the following four ways compared to previous works. First, we measured the effect of components such as the eyebrows and chin. And the accuracy of computer-based face recognition was compared to human-based face recognition according to facial components. Second, for computer-based recognition, facial components were automatically detected using the Adaboost algorithm and active appearance model (AAM), and user authentication was achieved with the face recognition algorithm based on principal component analysis (PCA). Third, we experimentally proved that the number of facial features (when including eyebrows, eye, nose, mouth, and chin) had a greater impact on the accuracy of human-based face recognition, but consistent inclusion of some feature such as chin area had more influence on the accuracy of computer-based face recognition because a computer uses the pixel values of facial images in classifying faces. Fourth, we experimentally proved that the eyebrow feature enhanced the accuracy of computer-based face recognition. However, the problem of occlusion by hair should be solved in order to use the eyebrow feature for face recognition.

In the contemporary global educational environment, the automatic assessment of students’ online engagement has garnered widespread attention. A substantial number of studies have demonstrated that facial expressions are a crucial indicator for measuring engagement. However, due to the asymmetry inherent in facial expressions and the varying degrees of deviation of students’ faces from a camera, significant challenges have been posed to accurate emotion recognition in the online learning environment. To address these challenges, this work proposes a novel VGG-SwishNet model, which is based on the VGG-16 model and aims to enhance the recognition ability of asymmetric facial expressions, thereby improving the reliability of student engagement assessment in online education. The Swish activation function is introduced into the model due to its smoothness and self-gating mechanism. Its smoothness aids in stabilizing gradient updates during backpropagation and facilitates better handling of minor variations in input data. This enables the model to more effectively capture subtle differences and asymmetric variations in facial expressions. Additionally, the self-gating mechanism allows the function to automatically adjust its degree of nonlinearity. This helps the model to learn more effective asymmetric feature representations and mitigates the vanishing gradient problem to some extent. Subsequently, this model was applied to the assessment of engagement and provided a visualization of the results. In terms of performance, the proposed method achieved high recognition accuracy on the JAFFE, KDEF, and CK+ datasets. Specifically, under 80–20% and 10-fold cross-validation (CV) scenarios, the recognition accuracy exceeded 95%. According to the obtained results, the proposed approach demonstrates higher accuracy and robust stability.

The aim of this study was to determine how the depressor supercilii (DS) connects to the levator labii superioris alaeque nasi (LLSAN) and inferior fibers of the orbicularis oculi (OOc INF) in the human midface. While grimacing, contraction of the DS with fibers connecting to the LLSAN and OOc INF can assist in pulling the medial eyebrow downward more than when these connecting fibers are not present. Contraction of these distinct connecting fibers between the DS and the LLSAN can also slightly elevate the nasal ala and upper lip. The DS was examined in 44 specimens of embalmed adult Korean cadavers. We found that the DS connected to the LLSAN or the OOc INF by muscle fibers or thin aponeuroses in 33 (75.0%) of the 44 specimens. The DS was connected to both the LLSAN and OOc INF by muscle fibers or aponeuroses and had no connection to either in 5 (11.4%) and 11 (25.0%) specimens, respectively. The DS was connected to the LLSAN by the muscle fibers and thin aponeuroses in 6 (13.6%) and 4 (9.1%) specimens, respectively. The DS was connected to the OOc INF by the muscle fibers and thin aponeuroses in 5 (11.4%) and 23 (52.3%) specimens, respectively. Our findings regarding the anatomical connections of the glabellar region DS to the midface LLSAN and OOc INF provide insights on the dynamic balance between the brow depressors such as the DS and brow-elevating muscle and contribute to understanding the anatomical origins of individual variation in facial expressions. These results can also improve the safety, predictability, and aesthetics of treatments for the glabellar region with botulinum toxin type A and can be helpful when performing electromyography.

This research features a new method for automatic face recognition robust to variations in lighting, facial expression and eyewear. The new algorithm named SKKUfaces (Sungkyunkwan University faces) employs PCA (Principal Component Analysis) and FLD (Fisher's Linear Discriminant) in series similarly to Fisherfaces. The fundamental difference is that SKKUfaces effectively eliminates, in the reduced PCA subspace, portions of the subspace that are responsible for variations in lighting and facial expression and then applies FLD to the resulting subspace. This results in superb discriminating power for pattern classification and excellent recognition accuracy. We also propose an efficient method to compute the between-class scatter and within-class scatter matrices for the FLD analysis. We have evaluated the performance of SKKUfaces using YALE and SKKU facial databases. Experimental results show that the SKKUface method is computationally efficient and achieves much better recognition accuracy than the Fisherface method [1] especially for facial images with variations in lighting and eyewear.

Most automatic facial expression analysis (AFEA) systems attempt to recognize facial expressions from data collected in a highly controlled environment with very high resolution frontal faces ( face regions greater than 200 x 200 pixels). However, in real environments, the face image is often in lower resolution and with head motion. It is unclear that the performance of AFEA systems for low resolution face images. The general approach to AFEA consists of 3 steps: face acquisition, facial feature extraction, and facial expression recognition. This paper explores the effects of different image resolutions for each step of facial expression analysis. The different approaches are compared for face detection, face data extraction and expression recognition. A total of five different resolutions of the head region are studied (288x384, 144x192, 72x96, 36x48, and 18Xx24) based on a widely used public database. The lower resolution images are down-sampled from the originals.

We describe a computer vision system for observing facial motion by using an optimal estimation optical flow method coupled with geometric, physical and motion-based dynamic models describing the facial structure. Our method produces a reliable parametric representation of the face's independent muscle action groups, as well as an accurate estimate of facial motion. Previous efforts at analysis of facial expression have been based on the facial action coding system (FACS), a representation developed in order to allow human psychologists to code expression from static pictures. To avoid use of this heuristic coding scheme, we have used our computer vision system to probabilistically characterize facial motion and muscle activation in an experimental population, thus deriving a new, more accurate, representation of human facial expressions that we call FACS+. Finally, we show how this method can be used for coding, analysis, interpretation, and recognition of facial expressions.

This paper proposes a novel natural facial expression recognition method that recognizes a sequence of dynamic facial expression images using the differential active appearance model (AAM) and k-NNS as follows. First, we use the differential-AAM features (DAFs) that are computed from the difference of the AAM parameters between an input face image and a reference face image. Second, we perform the manifold learning. Third, we recognize the facial expression of the input face image in the embedded feature space using sequence based k-NN, k-NNS. Since we use DAFs, we also propose an effective way of finding the neutral facial expression as kernel density approximation. Experimental results show that (1) the DAFs improves the facial expression recognition performance than the conventional AAM features by 20% and (2) the sequence-based k-nearest neighbors classifier provides a 95% of facial expression recognition performance on the facial expression database (FED06).

IntroductionUnderstanding how children recognize facial emotions is essential for explaining the development of social communication. During the COVID-19 pandemic, widespread mask-wearing partially occluded facial expressions, potentially influencing children's emotion recognition processes.MethodsThis study investigated age-related differences in emotion recognition accuracy and gaze fixation patterns in young children using eye-tracking technology. Participants included 36 three-year-olds and 45 five-year-olds. Children viewed facial expressions representing four emotions (joy, sadness, anger, and fear) presented in both masked and unmasked conditions while fixation duration and recognition accuracy were recorded.ResultsThe results showed significant effects of age, emotion type, and mask-wearing on emotion recognition accuracy. Overall recognition accuracy decreased under masked conditions. Eye-tracking analysis revealed that children allocated significantly greater visual attention to the eye region when faces were masked, whereas gaze distribution between the eyes and mouth was more balanced in unmasked conditions.DiscussionThese findings suggest that facial occlusion caused by mask-wearing alters visual attention strategies and emotion recognition processes in early childhood. The results provide important insights into how young children adapt their perceptual strategies when key facial cues are partially obscured.

This research aims to develop artificial intelligence (AI) algorithms in the context of facial recognition with a focus on increasing accuracy in difficult environmental conditions. Although facial recognition technology has made great progress, challenges such as poor lighting, variations in facial expressions, and head rotation are still problems that must be overcome. The research methodology involved collecting a wide dataset covering a wide variety of faces under various environmental conditions. This data is then processed and its features are extracted using computer image processing techniques. Furthermore, several deep neural network architectures, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), were developed, trained, and evaluated for face recognition tasks. The expected result is the development of an AI algorithm that is able to overcome challenges in facial recognition with higher accuracy than existing methods. In particular, significant improvements in facial recognition accuracy are expected especially under low lighting conditions and variations in facial expressions. This research has a major impact in a variety of security applications, such as border surveillance, building access control, and corporate security. With higher facial recognition accuracy, security risks can be significantly reduced, resulting in safer and more efficient security solutions. In conclusion, this research aims to bring innovation in facial recognition technology through advanced AI approaches, with the potential to improve security in various contexts.

There are many applications for systems coping with the problem of face localization and recognition, e.g. model-based video coding, security systems and mug shot matching. Due to variations in illumination, back-ground, visual angle and facial expressions, the problem of machine face recognition is complex. In this paper we present a robust approach for the extraction of facial regions and features out of color images. First, face candidates are located based on the color and shape information. Then the topographic grey-level relief of facial regions is evaluated to determine the position of facial features as eyes and month. Results are shown for two example scenes.

Facial expressions of pain play an important role in pain diagnostics and social interactions. Given the prominent impact of sex on various aspects of pain, it is not surprising that sex differences have also been explored regarding facial expressions of pain; however, with inconclusive findings. We aim to further investigate sex differences in facial expressions of pain by using a large, combined sample to maximize statistical power. Data from 7 previous studies of our group were merged, combining in total the data of 392 participants (male: 192, female: 200). All participants received phasic heat pain, with intensities being tailored to the individual pain threshold. Pain intensity ratings were assessed, and facial responses were manually analyzed using the Facial Action Coding. To compare facial and subjective responses between sexes, linear mixed-effects models were used, with study ID as a random effect. We found significant sex differences in facial responses, with females showing elevated facial responses to pain, although they received lower physical heat intensities (women had lower pain thresholds). In contrast, pain intensity ratings did not differ between sexes. Additionally, facial and subjective responses to pain were significantly associated across sexes, with females showing slightly stronger associations. Although variations in facial expressions of pain are very large even within each sex, our findings demonstrate that women facially communicate pain more intensively and with a better match to their subjective experience compared with men. This indicates that women might be better in using facial communication of pain in an intensity-discriminative manner.

Facial expressions are an ideal means of communicating one’s emotions or intentions to others. This overview will focus on human facial expression recognition as well as robotic facial expression generation. In the case of human facial expression recognition, both facial expression recognition on predefined datasets as well as in real-time will be covered. For robotic facial expression generation, hand-coded and automated methods i.e., facial expressions of a robot are generated by moving the features (eyes, mouth) of the robot by hand-coding or automatically using machine learning techniques, will also be covered. There are already plenty of studies that achieve high accuracy for emotion expression recognition on predefined datasets, but the accuracy for facial expression recognition in real-time is comparatively lower. In the case of expression generation in robots, while most of the robots are capable of making basic facial expressions, there are not many studies that enable robots to do so automatically. In this overview, state-of-the-art research in facial emotion expressions during human–robot interaction has been discussed leading to several possible directions for future research.

Deep convolution network based emotion analysis towards mental health care