FACIAL EMOTION RECOGNITION SYSTEM (FERS) USING MACHINE LEARNING

IntroductionFacial expressions play a crucial role in human emotion recognition and social interaction. Prior research has highlighted the significance of the eyes and mouth in identifying emotions; however, limited studies have validated these claims using robust biometric evidence. This study investigates the prioritization of facial features during emotion recognition and introduces an optimized approach to landmark-based analysis, enhancing efficiency without compromising accuracy.MethodsA total of 30 participants were recruited to evaluate images depicting six emotions: anger, disgust, fear, neutrality, sadness, and happiness. Eye-tracking technology was utilized to record gaze patterns, identifying the specific facial regions participants focused on during emotion recognition. The collected data informed the development of a streamlined facial landmark model, reducing the complexity of traditional approaches while preserving essential information.ResultsThe findings confirmed a consistent prioritization of the eyes and mouth, with minimal attention allocated to other facial areas. Leveraging these insights, we designed a reduced landmark model that minimizes the conventional 68-point structure to just 24 critical points, maintaining recognition accuracy while significantly improving processing speed.DiscussionThe proposed model was evaluated using multiple classifiers, including Multi-Layer Perceptron (MLP), Random Decision Forest (RDF), and Support Vector Machine (SVM), demonstrating its robustness across various machine learning approaches. The optimized landmark selection reduces computational costs and enhances real-time emotion recognition applications. These results suggest that focusing on key facial features can improve the efficiency of biometric-based emotion recognition systems without sacrificing accuracy.

The facial expressions recognition (FER) is crucial to many applications. As technology advances and our needs evolve, compound emotion recognition is becoming increasingly important, along with basic emotion recognition. In the literature, Although, FER can be conducted primarily using multiple sensors. However, research shows that using facial images/videos to recognize facial expressions is better because visual presentation can convey more efficiently. Among state-of-the-art methods for FER systems, to improve the accuracy of the basic and compound FER systems, detection of facial action units (AUs) must be combined to detect basic and compound facial expressions. State-of-the-art results show that machine learning and deep learning-based approaches are more potent than conventional FER approaches. This paper surveys various learning frameworks for facial emotion recognition systems for detecting basic and compound emotions using the diverse database and summarizing state-of-the-art results to give good understanding of impact of each learning framework used in FER systems.

The current study investigated if deficits in processing emotional expression affect facial identity processing and vice versa in children with autism spectrum disorder. Children with autism and IQ and age matched typically developing children classified faces either by emotional expression, thereby ignoring facial identity or by facial identity disregarding emotional expression. Typically developing children processed facial identity independently from facial expressions but processed facial expressions in interaction with identity. Children with autism processed both facial expression and identity independently of each other. They selectively directed their attention to one facial parameter despite variations in the other. Results indicate that there is no interaction in processing facial identity and emotional expression in autism spectrum disorder.

In this work, we have proposed enhancements that improve the performance of state-of-the-art facial emotion recognition (FER) systems. We believe that the changes in the positions of the fiducial points and the intensities capture the crucial information regarding the emotion of a face image. We propose the inputting of the gradient and the Laplacian of the input image together with the original into a convolutional neural network (CNN). These modifications help the network learn additional information from the gradient and Laplacian of the images. However, as shown by our results, the CNN in the existing state-of-the-art models is not able to extract this information from the raw images. In addition, we employ spatial transformer network to add robustness to the system against rotation and scaling. We have performed a number of experiments on two well known datasets, namely KDEF and FERplus. Our approach enhances the already high performance of the state-of-the-art FER systems by 3 to 5%.

Human emotion recognition has emerged as a critical component in the development of intelligent systems capable of understanding and responding to human emotional states. This comprehensive review examines the latest novel deep learning methods for emotion recognition, encompassing advances in neural architectures, multimodal fusion techniques, and innovative approaches to feature extraction and classification. The field has witnessed remarkable progress through the integration of sophisticated deep learning models including convolutional neural networks (CNNs), long short-term memory networks (LSTMs), transformer architectures, and generative adversarial networks (GANs). Recent breakthroughs have demonstrated exceptional performance improvements, with some methods achieving over 99% accuracy on benchmark datasets such as DEAP, SEED, and MAHNOB-HCI. This review analyzes the evolution from traditional machine learning approaches to state-of-the-art deep learning methodologies, examining their effectiveness across various modalities including electroencephalography (EEG), facial expressions, speech, and physiological signals.The paper provides a systematic analysis of fusion strategies, architectural innovations, and performance metrics while identifying key challenges and future research directions in the rapidly evolving landscape of emotion recognition technology.

Facial expressions are connected to experiencing emotions according to Facial-Feedback Theory of Emotion developed by Charles Darwin and William James [17]. Knowing how to read and interpret facial expressions helps us understand how human beings connect with one another in mobile-mediated communication (MMC) and improve our own abilities at managing emotions. There have been multiple types of research conducted that study and compare factors that influence facial emotion recognition in humans [10]. As one of the most prominent channel in human interactions, facial expression is a good indicator of a person’s emotions. Recent technological advancements have had a drastic impact on human interaction. Besides face-to-face chat, video chatting become more and more popular in the digital age. This research provides to understand how video, voice and music influence emotion perception in the MMC context among different user groups. In this research, we evaluated challenges and effectiveness between two primary emotions: happiness and sadness in video chatting. This study analyzed the emotional influences of voice and music on video chatting and its effectiveness on sensing others emotional energy. Participants were from 18–42 years old and 98% of participants are in between 18–32 years old. A series of visual stimuli were created with four models were from Gen Z; Millennial; Gen X; and Baby Boomer. In addition, the research examines the comfort level with different relationships (e.g. friends; acquaintances; and strangers) in the video chatting environment.

At the captivating nexus of technology and human emotions, we harness cutting-edge software to discern individuals' feelings through their facial expressions. This unique capability empowers us to provide customized recommendations for various forms of entertainment and enrichment, such as movies, music, books, and meditation practices. For instance, if a user appears sad, we can offer upbeat music to brighten their mood. Our ultimate objective is to enhance technology comprehension of emotions, enabling it to suggest content that resonates with people's emotional states. Through this integration of facial expressions and diversified suggestions, here aim is to cultivate a more user-friendly and supportive digital environment, fostering feelings of happiness and calmness. Exploring the captivating domain of facial emotion recognition and recommendation systems is the central emphasis of this endeavor. The primary ambition is to construct a framework capable of deciphering user emotions from facial cues. By using deep learning techniques such as the CNN algorithm which can be applied to facial images to process emotion recognition feature vectors with the categorization of individualized content. It attempts to form a balanced blend of advanced tech and boost user engagement, nurturing emotional connections in the digital domain.

Making human–computer interaction more organic and personalized for users essentially demands advancement in human emotion recognition. Emotions are perceived by humans considering multiple factors such as facial expressions, voice tonality, and information context. Although significant research has been conducted in the area of unimodal/multimodal emotion recognition in videos using acoustic/visual features, few papers have explored the potential of textual information obtained from the video utterances. Humans experience emotions through their audio-visual and linguistic senses, making it quintessential to take the latter into account. This paper outlines two different algorithms for recognizing multimodal emotional expressions in online videos. In addition to acoustic (speech), visual (facial), and textual (utterances) feature extraction using BERT, we utilize bidirectional LSTMs to capture the context between utterances. To obtain richer sequential information, we also implement a multi-head self-attention mechanism. Our analysis utilizes the benchmarking CMU multimodal opinion sentiment and emotion intensity (CMU-MOSEI) dataset, which is the largest dataset for sentiment analysis and emotion recognition to date. Our experiments result in improved F1 scores in comparison to the baseline models.KeywordMultimodalEmotionRecognitionCMU-MOSEIMulti-head attentionBERTContext-aware

Detecting human emotion based on facial expression is considered a hard task for the computer vision community because of many challenges such as the difference of face shape from a person to another, difficulty of recognition of dynamic facial features, low quality of digital images, etc. In this paper, we propose a face-sensitive convolutional neural network (FS-CNN) for human emotion recognition. The proposed FS-CNN is used to detect faces on large scale images then analyzing face landmarks to predict expressions for emotion recognition. The FS-CNN is composed form two stages, patch cropping, and convolutional neural networks. The first stage is used to detect faces in high-resolution images and crop the face for further processing. The second stage is a convolutional neural network used to predict facial expression based on landmarks analytics, it was applied on pyramid images to process scale invariance. The proposed FS-CNN was trained and evaluated on the UMD Faces dataset. High performance was achieved with a mean average precision of about 95%.

Emotional state recognition is an important component for efficient human-computer interaction. Most existing works address this problem using 2D features, but they are sensitive to head pose, clutter, and variations in lighting conditions. The general 3D based methods only consider geometric information for feature extraction. In this paper, we present a real 3D visual features based method for human emotion recognition. 3D geometric information plus colour/density information of the facial expressions are extracted by 3D Gabor library to construct visual feature vectors. The filter's scale, orientation, and shape of the library are specified according to the appearance patterns of the 3D facial expressions. An improved kernel canonical correlation analysis (IKCCA) algorithm is proposed for final decision. From training samples, the semantic ratings that describe the different facial expressions are computed by IKCCA to generate a seven dimensional semantic expression vector. It is applied for learning the correlation with different testing samples. According to this correlation, we estimate the associated expression vector and perform expression classification. From experiment results, our proposed method demonstrates impressive performance.

Humans have traditionally found it simple to identify emotions from facial expressions, but it is far more difficult for a computer system to do the same. The social signal processing subfield of emotion recognition from facial expression is used in a wide range of contexts, particularly for human-computer interaction. Automatic emotion recognition has been the subject of numerous studies, most of which use a machine learning methodology. The recognition of simple emotions like anger, happiness, contempt, fear, sadness, and surprise, however, continues to be a difficult topic in computer vision. Deep learning has recently drawn increased attention as a solution to a variety of practical issues, including emotion recognition. In this study, we improved the convolutional neural network technique to identify 7 fundamental emotions and evaluated several preprocessing techniques to demonstrate how they affected the CNN performance. This research focuses on improving facial features and expressions based on emotional recognition. By identifying or recognising facial expressions that elicit human responses, it is possible for computers to make more accurate predictions about a person's mental state and to provide more tailored responses. As a result, we examine how a deep learning technique that employs a convolutional neural network might improve the detection of emotions based on facial features (CNN). Multiple facial expressions are included in our dataset, which consists of about 32,298 photos for testing and training. The preprocessing system aids in removing noise from the input image, and the pretraining phase aids in revealing face detection after noise removal, including feature extraction. As a result, the existing paper generates the classification of multiple facial reactions like the seven emotions of the facial acting coding system (FACS) without using the optimization technique, but our proposed paper reveals the same seven emotions of the facial acting coding system.

Optimized machine learning approaches for identifying vertical temperature gradient on ballastless track in natural environments

Patients with amygdala lesions were found to be impaired in recognizing the fear emotion both from face and from music. In patients with Parkinson's disease (PD), impairment in recognition of emotions from facial expressions was reported for disgust, fear, sadness and anger, but no studies had yet investigated this population for the recognition of emotions from both face and music. The ability to recognize basic universal emotions (fear, happiness and sadness) from both face and music was investigated in 24 medicated patients with PD and 24 healthy controls. The patient group was tested for language (verbal fluency tasks), memory (digit and spatial span), executive functions (Similarities and Picture Completion subtests of the WAIS III, Brixton and Stroop tests), visual attention (Bells test), and fulfilled self-assessment tests for anxiety and depression. Results showed that the PD group was significantly impaired for recognition of both fear and sadness emotions from facial expressions, whereas their performance in recognition of emotions from musical excerpts was not different from that of the control group. The scores of fear and sadness recognition from faces were neither correlated to scores in tests for executive and cognitive functions, nor to scores in self-assessment scales. We attributed the observed dissociation to the modality (visual vs. auditory) of presentation and to the ecological value of the musical stimuli that we used. We discuss the relevance of our findings for the care of patients with PD.

The digital animation process is a complex endeavour requiring professional animators to acquire substantial expertise and technique through years of study and practice. Particularly, facial animation, where virtual humans express specific mental states or emotions with a desire for realism, is further complicated by the “Uncanny Valley” phenomenon. In this context, it is posited that pre-validated facial expressions for certain emotions could serve as references for the novice or inexperienced animators during the facial animation and posing process of their virtual humans using morph targets, also known as blend shapes or shape keys. This research presents a comparative study between two Facial Expression Recognition (FER) systems that employ pre-trained models for facial recognition applied to emotion recognition in virtual humans. Given that these systems were not designed or trained for this particular purpose but for facial recognition in real humans, this study aims to investigate their level of applicability in scenarios where virtual humans are used instead of real humans. This assessment is a critical step towards evaluating the feasibility of integrating FER models as part of a support tool for facial animation and the posing of virtual humans. Through this investigation, this research provides evidence of the reliability of applying the FER library and Deepface systems for emotion recognition in virtual humans, contributing to investigating new ways to enhance the digital animation process and overcoming the inherent complexities of facial animation.

Performance optimization of wireless networks is typically complicated because of high computational complexity and dynamic channel conditions. Considering a specific case, the recent introduction of intelligent reflecting surface (IRS) can reshape the wireless channels by controlling the scattering elements' phase shifts, namely, passive beamforming. However, due to the large size of scattering elements, the IRS's beamforming optimization becomes intractable. In this article, we focus on machine learning (ML) approaches for complex optimization problems in wireless networks. ML approaches can provide flexibility and robustness against uncertain and dynamic systems. However, practical challenges still remain due to slow convergence in offline training or online learning. This motivated us to design a novel optimization-driven ML framework that exploits the efficiency of model-based optimization and the robustness of model-free ML approaches. Splitting the control variables into two parts allows one part to be updated by the outer loop ML approach while the other part is solved by the inner loop optimization. The case study in IRS-assisted wireless networks confirms that the optimization-driven ML framework can improve learning efficiency and the reward performance significantly compared to conventional model-free ML approaches.