Electroencephalogram-based emotion recognition using involvement indexes: A machine-learning approach

IntroductionFacial expressions play a crucial role in human emotion recognition and social interaction. Prior research has highlighted the significance of the eyes and mouth in identifying emotions; however, limited studies have validated these claims using robust biometric evidence. This study investigates the prioritization of facial features during emotion recognition and introduces an optimized approach to landmark-based analysis, enhancing efficiency without compromising accuracy.MethodsA total of 30 participants were recruited to evaluate images depicting six emotions: anger, disgust, fear, neutrality, sadness, and happiness. Eye-tracking technology was utilized to record gaze patterns, identifying the specific facial regions participants focused on during emotion recognition. The collected data informed the development of a streamlined facial landmark model, reducing the complexity of traditional approaches while preserving essential information.ResultsThe findings confirmed a consistent prioritization of the eyes and mouth, with minimal attention allocated to other facial areas. Leveraging these insights, we designed a reduced landmark model that minimizes the conventional 68-point structure to just 24 critical points, maintaining recognition accuracy while significantly improving processing speed.DiscussionThe proposed model was evaluated using multiple classifiers, including Multi-Layer Perceptron (MLP), Random Decision Forest (RDF), and Support Vector Machine (SVM), demonstrating its robustness across various machine learning approaches. The optimized landmark selection reduces computational costs and enhances real-time emotion recognition applications. These results suggest that focusing on key facial features can improve the efficiency of biometric-based emotion recognition systems without sacrificing accuracy.

Electrocardiogram captures the electrical activity of the heart. The signal obtained can be used for various purposes such as emotion recognition, heart rate measuring and the main one, cardiac disease diagnosis. But ECG analysis and classification require experienced specialists once it presents high variability and suffers interferences from noises and artefacts. With the increase of data amount on long term records, it might lead to long term dependencies and the process become exhaustive and error prone. Automated systems associated with signal processing techniques aim to help on these tasks by improving the quality of data, extracting meaningful features, selecting the most suitable and training machine learning models to capture and generalize its behaviour. This review brings a brief stage sense of how data flows into these approaches and somewhat techniques are most used. It ends by presenting some of the countless applications that can be found in the research community.

Face identification has been around for a while. Moving forward, it will be possible to assess employee emotion based on facial expressions, internal experiences, and recordings in the form of video, electric signals, or images. In order for cutting-edge artificial intelligence systems to replicate and forecast face reactions, human emotion recognition is critically required. This might help people make informed decisions about things like purpose determination, offering advertising, and security-related dangers. The human eye can easily identify emotions in images or videos, but computers find it more challenging to do so, necessitating the employment of different image processing techniques for feature extraction. Several machine learning approaches are suitable for this task.

Emotion and mood recognition plays a key role in human-robot interaction, especially in the context of socially assistive robotics. Mood-aware robots could be useful as companions and social assistants for elders and people affected by depression and other mood disorders. An interesting option for continuously tracking a user's mood is the use of wearable and mobile devices. However, the classification of the mood from physiological and kinematics data is still a challenge, due to intersubjects differences: on one hand, one-fits-all classification approaches usually achieve lower accuracy than person-specific methods; on the other hand, personalized models require in general a large amount of data from a single subject to be trained and, therefore, becomes effective after long periods of acquisition. In this paper, we propose a deep learning approach for mood recognition from a publicly available dataset that includes a gyroscope, accelerometer, and heart-rate data. We propose the use of long-short term memory networks (LSTM), testing them both as classifiers and as features extractors in hybrid models. We compared their performances both against and in conjunction with traditional machine learning approaches, namely support vector machines (SVM) and Gaussian mixture models (GMM). We also consider transfer learning strategies to reduce the amount of personal data needed to train the model. Our results show that the use of LSTMs significantly improves the classification accuracy with respect to machine learning approaches, especially if employed as feature extractors and combined with SVM. However, we observed that transfer learning does not achieve significant results in boosting the training of a personalized model.

There are various techniques for extracting relevant conclusions from various types of data the efficient approach that combines the power of data analysis, with intelligence and machine learning is smart web mining which enables the extraction of diverse data forms including text documents, videos and audio files and images effectively and efficiently. This study delves into the techniques and resources employed in web mining to accurately assess and comprehend data. Most web data is in the form of text. They are commonly fetched with algorithms like topic modelling, sentiment analysis, and other machine learning approaches known as natural language processing (NLP). These sources can appear to be inactive and unstructured, but indeed, these methods and other computer language model frameworks can decipher important information from them. For image data, algorithms for object detection use computer vision methods such as convolutional neural networks (CNN). While trying to detect patterns and information, faces or objects are identified and classified. Emotion, speaker identification, and automatic speech recognition are some of the applications of mining audio with neural networks, spectral, and speech to text analysis. The analysis of motion and interaction of objects in films is done using a fusion of visual and auditory analysis with guided methods. In building such frameworks, various algorithms giving the ability to process data from different sources are integrated. In handling the huge and heterogeneous information on the internet, intelligent web mining systems use the combination of the cloud computing service and computers that were built on the decentralized databases. Over and above sophisticated machine learning algorithms are trained with the datasets when the need for accuracy and flexibility is considered as the unsupervised learning technique reveals the hidden associations or patterns. This study highlights smart web mining's potential as a transformative tool for extracting insights from various data types, paving the way for innovative applications in a data-driven world.

The facial expressions recognition (FER) is crucial to many applications. As technology advances and our needs evolve, compound emotion recognition is becoming increasingly important, along with basic emotion recognition. In the literature, Although, FER can be conducted primarily using multiple sensors. However, research shows that using facial images/videos to recognize facial expressions is better because visual presentation can convey more efficiently. Among state-of-the-art methods for FER systems, to improve the accuracy of the basic and compound FER systems, detection of facial action units (AUs) must be combined to detect basic and compound facial expressions. State-of-the-art results show that machine learning and deep learning-based approaches are more potent than conventional FER approaches. This paper surveys various learning frameworks for facial emotion recognition systems for detecting basic and compound emotions using the diverse database and summarizing state-of-the-art results to give good understanding of impact of each learning framework used in FER systems.

Music is ubiquitous in every corner of the world. Fundamental mechanisms for how a listener perceives emotions in music as well as the way in which the emotional goal of a composer is expressed through music have been widely explored. Not only humans but also machines can recognize emotions in music once they are trained. As a result, various machine learning approaches are tested by previous researchers for their capability in music emotion recognition. To identify the current status of research and the research gaps in the above domain, we conducted a systematic literature review, through which we investigated several key aspects including types of music, acoustic features, feature extraction mechanisms, classification algorithms, and their performance. Six electronic databases were searched for studies published until 2018. Initial set of studies comprised of 198 studies from which 14 were selected for detailed analysis. Findings revealed that a considerable number of studies have used western music whereas other cultural-specific music still remains to be explored. Acoustic features pertaining to pitch, intensity, timbre, tempo, rhythm, melody, and harmony have been commonly used. While Support Vector Machine, Naive Bayes, and k-Nearest Neighbor are among the frequently used standard classifiers, Fuzzy classifiers and ensemble learning also have been attempted. The discrepancies among the classifier performances reported in previous studies could be partially attributed to the differences in the key aspects we have considered. The study makes a significant contribution by providing a comprehensive and up-to-date review of the previous attempts made in the selected domain.

This article explores the integration of emotional intelligence (EI) into AI voice assistants, examining techniques for emotion recognition from speech, adaptive response generation, and the impact on user experience. It discusses key components including acoustic feature analysis, machine learning approaches, and multimodal systems for emotion detection. The article also addresses ethical considerations such as privacy concerns, potential manipulation, and technical challenges in achieving robust, real-time, and culturally-adaptive EI systems. Applications in mental health support and customer service are examined, highlighting both the potential benefits and necessary precautions. Through a comprehensive review of recent research, this work aims to contribute to the responsible development of emotionally intelligent AI voice assistants that enhance human-AI interaction while prioritizing user well-being.

Most Facial Expression Recognition (FER) systems rely on machine learning approaches that require large databases for an effective training. As these are not easily available, a good solution is to augment the databases with appropriate data augmentation (DA) techniques, which are typically based on either geometric transformation or oversampling augmentations (e.g., generative adversarial networks (GANs)). However, it is not always easy to understand which DA technique may be more convenient for FER systems because most state-of-the-art experiments use different settings which makes the impact of DA techniques not comparable. To advance in this respect, in this paper, we evaluate and compare the impact of using well-established DA techniques on the emotion recognition accuracy of a FER system based on the well-known VGG16 convolutional neural network (CNN). In particular, we consider both geometric transformations and GAN to increase the amount of training images. We performed cross-database evaluations: training with the "augmented" KDEF database and testing with two different databases (CK+ and ExpW). The best results were obtained combining horizontal reflection, translation and GAN, bringing an accuracy increase of approximately 30%. This outperforms alternative approaches, except for the one technique that could however rely on a quite bigger database.

Human emotions can be expressed in many ways through facial expressions, speech, actions, and in textual form. Emotions play an important role in determining the human’s state and influence their lives in decision-making, behaviour and interaction. With the evolution of many advances in technology, researchers are able to detect the emotion of humans through facial expressions, speech, and text. Much advanced and efficient research work has been done in the field of emotion recognition through facial expression and audio, but still, there are not great works on emotion detection from text. Detecting emotions from text has wide applications in stock markets, business, decision-making, analyzing the view of people on any topic through social media conversations, tweets, blogs, and articles. Our work is focused on the detection of emotions from the text. We propose different approaches for detecting emotions through text. We implemented different approaches like the lexicon-based approach, the supervised machine learning approach with the Naïve-bayes algorithm, and the unsupervised machine learning approach with semantic similarities. After the consideration of the cons of all the approaches and the accuracies derived from our work, it is evident that the unsupervised machine learning approach with semantic similarities is the best approach with an accuracy of 78.5%.

A systematic review on machine-learning strategies for improving generalization in electroencephalography-based emotion classification was realized. In particular, cross-subject and cross-session generalization was focused. In this context, the non-stationarity of electroencephalographic (EEG) signals is a critical issue and can lead to the Dataset Shift problem. Several architectures and methods have been proposed to address this issue, mainly based on transfer learning methods. In this review, 449 papers were retrieved from the Scopus, IEEE Xplore and PubMed databases through a search query focusing on modern machine learning techniques for generalization in EEG-based emotion assessment. Among these papers, 79 were found eligible based on their relevance to the problem. Studies lacking a specific cross-subject or cross-session validation strategy, or making use of other biosignals as support were excluded. On the basis of the selected papers’ analysis, a taxonomy of the studies employing Machine Learning (ML) methods was proposed, together with a brief discussion of the different ML approaches involved. The studies reporting the best results in terms of average classification accuracy were identified, supporting that transfer learning methods seem to perform better than other approaches. A discussion is proposed on the impact of (i) the emotion theoretical models and (ii) psychological screening of the experimental sample on the classifier performances.

Neural networks are increasingly able to outperform traditional machine learning and filtering approaches in classification tasks. However, with the rise in their popularity, many unknowns still exist when it comes to the internal learning processes of the networks in terms of how they make the right decisions for prediction. As a result, in this work, different attention modules integrated into a convolutional neural network coupled with an attention-guided strategy were examined for facial emotion recognition performance. A custom attention block, AGFER, was developed and evaluated against two other well-known modules of squeeze–excitation and convolution block attention modules and compared with the base model architecture. All models were trained and validated using a subset from the OULU-CASIA database. Afterward, cross-database testing was performed using the FACES dataset to assess the generalization capability of the trained models. The results showed that the proposed attention module with the guidance strategy showed better performance than the base architecture while maintaining similar results versus other popular attention modules. The developed AGFER attention-integrated model focused on relevant features for facial emotion recognition, highlighting the efficacy of guiding the model during the integral training process.

Emotional awareness has been the subject of research in the field of Human-Computer Interaction in recent years. Computers have become an integral part of human life. Users need human-like interactions to better communicate with computers. Many researchers have done so be interested in emotional recognition and classification using a variety of sources. For recognizing emotions, we proposed a machine learning and deep learning-based approach namely CNLSTM in video expressions. The hybrid approach is employed to dissect the video data and perform recognition of emotions. The CNN extracts the features from the corresponding frames in the video and LSTM is used to explore the temporal dynamics of the video. To validate the proposed approach, two datasets namely CK+, JAFFE, and Aff-wild2 have been taken. Different experiments have been performed to validate the results of the proposed approach. The experimental results show that our proposed approach achieves accuracy in the angry mode of the CK+ dataset. Also, the proposed model shows better results in unbalanced data.

Emotion detection from video, audio, and text has emerged as a vital area of research within the fields of artificial intelligence and human-computer interaction. As digital communication increasingly integrates multiple modalities, understanding human emotions through these various channels has become essential for enhancing user experience, improving mental health diagnostics, and advancing affective computing technologies. This paper presents a comprehensive overview of the methodologies and frameworks developed for detecting emotions from video, audio, and text inputs, highlighting the synergies and challenges of multimodal emotion recognition systems. The paper begins by discussing the significance of each modality in emotion detection. Video analysis leverages facial expressions, body language, and gestures, employing computer vision techniques to extract key features that indicate emotional states. Audio processing focuses on vocal characteristics, such as tone, pitch, and speech patterns, utilizing signal processing and machine learning algorithms to interpret the emotional nuances conveyed through speech. Text analysis, on the other hand, relies on natural language processing (NLP) techniques to assess sentiment and emotional context from written language, considering both syntactic and semantic factors. By integrating these three modalities, the proposed systems can achieve more accurate and robust emotion recognition, reflecting the complexity of human emotional expression. Moreover, the paper explores the challenges faced in multimodal emotion detection, including data synchronization, feature extraction, and the need for large, annotated datasets that represent diverse emotional expressions across different cultures and contexts. The integration of machine learning and deep learning approaches is examined, showcasing how these technologies enhance the effectiveness of emotion detection systems. Recent advancements, such as the use of transformer architectures and attention mechanisms, have shown promise in capturing the relationships between modalities and improving the overall classification accuracy. Finally, this research emphasizes the potential applications of multimodal emotion detecti

Facial emotion recognition (FER) is a topic that has gained interest over the years for its role in bridging the gap between Human and Machine interactions. This study explores the potential of real time FER modelling, to be integrated in a closed loop system, to help in treatment of children suffering from Autism Spectrum Disorder (ASD). The aim of this study is to show the differences between implementing Traditional machine learning and Deep learning approaches for FER modelling. Two classification approaches were taken, the first approach was based on classic machine learning techniques using Histogram of Oriented Gradients (HOG) for feature extraction, with a k-Nearest Neighbor and a Support Vector Machine model as classifiers. The second approach uses Transfer Learning based on the popular “Alex Net” Neural Network architecture. The performance of the approaches was based on the accuracy of randomly selected validation sets after training on random training sets of the Oulu-CASIA database. The data analyzed shows that traditional machine learning methods are as effective as deep neural net models and are a good compromise between accuracy, extracted features, computational speed and costs.