Design and Implementation of an Electroencephalography System for Emotion Recognition

Purpose: Listening to music has a great impact on people's emotions and would change brain activity. In other words, music-induced emotions are trackable in electrical brain activities. Therefore, Electroencephalography can be a suitable tool to detect these induced emotions. The present study attempted to use electroencephalography in to recognize four types of emotions (happy, relaxing, stressful, and sad) induced in response to listening to music excerpts, using three classifiers.
 Materials and Methods: In this empirical study, electroencephalography signals were collected from 20 participants, as they were listening to pieces of selected music. The collected data were then pre-processed, and 28 linear and nonlinear features for recognizing the aforementioned emotions were extracted. Feature-space components were then reduced through a principal components analysis. Finally, the first ten components of feature-space were used as input for three classifiers based on Neural Network (NN), K-Nearest Neighbors (KNN), and Support Vector Machine (SVM) algorithms to identify the induced emotions.
 Results: The outputs showed that the suggested method was well capable of emotion recognition. Evaluating the music excerpts, on the self-assessment manikin scale, demonstrated that the labeling of the music tracks was accurate. The highest accuracy found among NN, KNN, and SVM algorithms were %84, %84, and %89 for happy emotions, respectively.
 Conclusion: The findings of this study provide useful insights into emotion classification and brain behavior related to induced emotion extraction. Happiness was the most recognizable emotion and the support vector machine had the highest performance among the classifiers. In the end, the outcomes of the proposed method demonstrate that this system is better than the previous research in EEG-based emotion recognition.

Current literature suggests that theory of mind (ToM) and recognition of facial emotion are impaired in behavioral variant frontotemporal dementia (bvFTD). In contrast, studies suggest that ToM is spared in Alzheimer disease (AD). However, there is controversy whether recognition of emotion in faces is impaired in AD. This study challenges the concepts that ToM is preserved in AD and that recognition of facial emotion is impaired in bvFTD. ToM, recognition of facial emotion, and identification of emotions associated with video vignettes were studied in bvFTD, AD, and normal controls. ToM was assessed using false-belief and visual perspective-taking tasks. Identification of facial emotion was tested using Ekman and Friesen's pictures of facial affect. After adjusting for relevant covariates, there were significant ToM deficits in bvFTD and AD compared with controls, whereas neither group was impaired in the identification of emotions associated with video vignettes. There was borderline impairment in recognizing angry faces in bvFTD. Patients with AD showed significant deficits on false belief and visual perspective taking, and bvFTD patients were impaired on second-order false belief. We report novel findings challenging the concepts that ToM is spared in AD and that recognition of facial emotion is impaired in bvFTD.

EEG-based emotion identification from nerve conduction mechanisms: A gustatory-emotion coupling model combined with multiblock attention module

Electroencephalographic (EEG)-based emotion recognition has received increasing attention in the field of human-computer interaction (HCI) recently, there however remain a number of challenges in building a generalized emotion recognition model, one of which includes the difficulty of an EEG-based emotion classifier trained on a specific stimulus to handle other stimuli. Little attention has been paid to this issue. The current paper is to study this issue and determine the feasibility of coping with this challenge using feature selection. 12 healthy volunteers were emotionally elicited when watching the video clip. Power spectral density (PSD) and brain asymmetry (BAY) were extracted as features. Support vector machine (SVM) classifier was then examined under within-stimulus conditions (samples extracted from one video were sent to both training set and testing set) and cross-stimulus conditions (samples extracted from one video were merely sent to one set, training set or testing set alternatively). The within-stimulus 5-class classification performed fairly well (accuracy: 93.31% for PSD and 85.39% for BAY). Cross-stimulus classification, however, deteriorated to low levels (46.22% and 46.2% accordingly). Trained and tested with the most robust feature subset selected by SVM-recursive feature elimination (RFE), the mean 5-class performance of cross-stimulus classifier was significantly improved to 68.89% and 64.44% for PSD and BAY respectively. These results suggest that cross-stimulus emotion recognition is reasonable and feasible with proper methods and brings EEG-based emotion recognition models closer to being able to discriminate emotion states in practical application.

In human interactions, emotion recognition is crucial. For this reason, the topic of computer-vision approaches for automatic emotion recognition is currently being extensively researched. Processing multi-channel electroencephalogram (EEG) information is one of the most researched methods for automatic emotion recognition. This paper presents a new model for an affective computing-driven Quality of Experience (QoE) prediction. In order to validate the proposed model, a publicly available dataset is used. The dataset contains EEG, ECG, and respiratory data and is focused on a multimedia QoE assessment context. The EEG data are retained on which the differential entropy and the power spectral density are calculated with an observation window of three seconds. These two features were extracted to train several deep-learning models to investigate the possibility of predicting QoE with five different factors. The performance of these models is compared, and the best model is optimized to improve the results. The best results were obtained with an LSTM-based model, presenting an F1-score from 68% to 78%. An analysis of the model and its features shows that the Delta frequency band is the least necessary, that two electrodes have a higher importance, and that two other electrodes have a very low impact on the model's performances.

Advancements in neuroscience and deep learning have significantly enhanced bio-signal-based emotion recognition, a critical component in Brain-Machine Interface (BMI) applications for healthcare, human-computer interaction, and human-AI assistant communication. Former studies have proposed Manual Mapping electrode matrices and employing Convolutional Neural Networks (CNNs) to recognize spatial EEG activities. However, this Manual Mapping of EEG electrodes onto matrix grids limits spatial precision and introduces inefficiencies. This study proposes automated channel mapping methods of Orthographic Projection and Stereographic Projection to address these challenges, using Differential Entropy and Power Spectral Density with Linear Dynamical Systems as features. A 3-branch multiscale CNN was trained on open-source dataset, employing a 5-fold cross-classification approach. Experimental results demonstrate that higher-resolution grids (16×16, 24×24) with automated projections significantly outperform Manual Mappings, achieving up to a 4.06% improvement in classification accuracy (p < 0.05). This result indicates that enhancing spatial precision of EEG data improves emotion classification, establishing automated spatial mapping as an advancement in EEG-based emotion recognition.Clinical Relevance-Advancement in emotion classification accuracy can facilitate more reliable diagnostic tools and personalized therapeutic interventions for mental health disorders, such as depression and anxiety.

Beyond BCI—Validating a wireless, consumer-grade EEG headset against a medical-grade system for evaluating EEG effects of a test anxiety intervention in school

Employing PCA and t-statistical approach for feature extraction and classification of emotion from multichannel EEG signal

Electroencephalography (EEG) signals are physiological signals that directly reflect the brain’s electrical activity, holding significant research value in the field of emotion recognition. However, existing methods still face certain limitations in jointly modeling temporal, spatial, and frequency-domain features. To address this, this article proposes an EEG-based emotion recognition model that integrates a three-dimensional convolutional neural network (3DCNN) with a bidirectional long short-term memory network (BiLSTM), referred to as 3DC-BiL. This approach utilizes 3DCNN to extract spatio-temporal features across different frequency bands, and then leverages BiLSTM to enhance temporal modeling capabilities, achieving collaborative fusion of multidimensional features. On the DEAP dataset for binary emotion classification, the proposed model achieves average accuracies of 98.60% and 98.66% on the arousal and valence dimensions, respectively. On the SEED dataset for three-class emotion classification, it achieves an average accuracy of 99.01%, outperforming various mainstream emotion recognition models in recent years. The experimental results demonstrate that the proposed model excels in EEG feature extraction and high-precision emotion classification, providing a new perspective and practical foundation for the research of EEG-based emotion recognition systems.

Emotion recognition using physiological signals has been a special topic frequently discussed by researchers and practitioners in the past decade. However, the use of SpO2 and Pulse rate signals for emotion recognitionisvery limited and the results still showed low accuracy. It is due to the low complexity of SpO2 and Pulse rate signals characteristics. Therefore, this study proposes a Multiscale Entropy and Multiclass Fisher’s Linear Discriminant Analysis for feature extraction and dimensional reduction of these physiological signals for improving emotion recognition accuracy in elders. In this study, the dimensional reduction process was grouped into three experimental schemes, namely a dimensional reduction using only SpO2 signals, pulse rate signals, and multimodal signals (a combination feature vectors of SpO2 and Pulse rate signals). The three schemes were then classified into three emotion classes (happy, sad, and angry emotions) using Support Vector Machine and Linear Discriminant Analysis Methods. The results showed that Support Vector Machine with the third scheme achieved optimal performance with an accuracy score of 95.24%. This result showed a significant increase of more than 22%from the previous works.

A motor imagery vs. rest dataset with low-cost consumer grade EEG hardware

Electroencephalogram (EEG) provides valuable information about brain activities and states in a non-invasive way, making it a crucial research area in human-computer interaction (HCI). With the rapid advancement of artificial intelligence (AI) technologies, EEG systems have increasingly harnessed the power of AI for various clinical, entertainment, and social interaction applications. For example, sleep staging systems combine EEG signals with deep learning to assist physicians in the rapid diagnosis of sleep disorders. Driver monitoring systems employ EEG-based deep neural networks (DNNs) to accurately detect driver fatigue, thereby reducing the risk of car accidents. Additionally, robotic arm control systems use DNNs to translate human thoughts, as reflected by EEG signals, into control signals, enabling disabled individuals to perform basic tasks such as drinking water or moving objects. Despite the significant progress driven by AI, models, particularly those based on deep learning, remain largely unexplainable due to their black-box nature. This lack of interpretability poses challenges in understanding and trusting the AI's decisions. Furthermore, these models are susceptible to both intentional and unintentional attacks, raising serious concerns about their robustness and reliability. Addressing these issues is crucial for AI's widespread adoption and safe deployment in EEG systems. Researchers are actively exploring methods to enhance the interpretability and robustness of AI models, ensuring they can provide reliable and transparent support in critical applications. As the field evolves, these advancements will be pivotal in realizing the full potential of AI-enhanced EEG systems for improving human life across various domains. A comprehensive literature review on interpretable and robust AI techniques for EEG systems is presented in this thesis. It begins with an introduction to the background knowledge of EEG signals. Next, it proposes a taxonomy of interpretability, categorizing it into three types: backpropagation, perturbation, and rule-based methods. Additionally, it categorizes robustness based on undesirable factors into four classes: noise and artifacts, human variability, data acquisition instability, and adversarial attacks. The literature review includes detailed analyses and comparisons for each category. Finally, it identifies several critical challenges for interpretable and robust AI in EEG systems and discusses their future directions. This literature review aims to guide researchers in understanding this field's latest advancements and future trends. This thesis introduces a novel framework called the Hybrid Attention EEG Sleep Staging (HASS) Framework, designed for cross-subject EEG sleep staging tasks. HASS employs a spatio-temporal attention mechanism to adaptively assign weights to inter-channel and intra-channel EEG segments based on the spatio-temporal relationships of the brain during different sleep stages. Experimental results on the MASS and ISRUC datasets demonstrate that HASS can significantly improve typical sleep staging networks. HASS addresses the difficulties of capturing the spatial-temporal relationships of EEG signals during sleep staging under cross-subject scenarios and holds promise for improving the accuracy and reliability of sleep assessment in both clinical and research settings. Furthermore, this thesis introduces EENED and GlepNet, novel EEG-based architectures for neural epilepsy detection. To address the challenge of learning robust global-local representations in epilepsy signals, EENED/GlepNet combines temporal convolutional layers with a multi-head attention mechanism. This approach enhances the performance of epilepsy diagnosis, potentially improving patient outcomes. The utilization of Grad-CAM for interpretability further elevates its clinical value, allowing healthcare professionals to validate and visually understand the model's diagnostic process. This advancement not only promises improved epilepsy detection but also contributes significantly to neuropsychological research and the application of machine learning in healthcare. Additionally, an Interpretability-guided Channel Selection (ICS) framework is proposed for the EEG driver drowsiness detection task. ICS provides a two-stage training strategy to select the key contributing channels with interpretability guidance progressively. ICS trains a teacher network in the first stage using full-head channel EEG data. It then applies class activation mapping (CAM) to the trained teacher model to highlight the high-contributing EEG channels and proposes a channel voting scheme to select the top $N$ contributing channels. In the second stage, ICS trains a student network with the selected channels of EEG data for driver drowsiness detection. Experiments on a public dataset demonstrate that our method significantly improves the performance of cross-subject driver drowsiness detection. Finally, by adopting relational thinking theory to transform raw EEG signals into probabilistic graphs, this thesis improves the decoding performance for EEG emotion classification tasks. The proposed method, the relational probabilistic graph convolutional network (RPGCN), effectively models variations in potential emotional states. RPGCN considers relationships among EEG channels and provides interpretability by explaining recognition results consistent with cognitive neuroscience findings. Extensive experiments demonstrate that RPGCN significantly outperforms state-of-the-art approaches for EEG-based emotion recognition. The interpretable modeling of EEG signals opens new possibilities for integrating brain activity analysis to enable more intelligent and personalized human-computer interaction.

Conventional EEG devices have limitations for the use of Brain-Computer Interface (BCI) because they are uncomfortable to wear in daily life. Since most smartphone users use earphones, a novel Earphone-shaped EEG device, which measures EEG signals in the ear canal while maintaining functions of the earphone, can be powerful tools for BCI. In this report, the attention state recorded from in-ear EEG was discriminated from the resting state to use simple application of one-button menu selection. Power spectral densities (PSD) in eye-closed state, eye-open state, and attention state were compared using autoregressive (AR) Burg method. Using selected features from Fisher ratio, attention state was successfully classified from resting state with support vector machine (SVM). Based on this study, prototypes for stable recording and sound delivery are developing and real-time BCI application using earphone-shaped EEG device will be researched.

Familiarity is a crucial subjectivity issue in music perception that is often overlooked in neural correlate studies and emotion recognition research. We investigated the effects of familiarity to brain activity based on electroencephalogram (EEG). In our research, we focused on self-reporting and continuous annotation based on the hypothesis that the emotional state in music experiencing is subjective and changes over time. Our methodology allowed subjects to select 16 MIDI songs, comprised of 8 familiar and 8 unfamiliar songs. We found evidence that music familiarity induces changes in power spectral density and brain functional connectivity. Furthermore, the empirical results suggest that using songs with low familiarity could slightly enhance EEG-based emotion classification performance with fractal dimension or power spectral density feature extraction algorithms and support vector machine, multi-layer perceptron or C4.5 classifiers. Therefore, unfamiliar songs would be most appropriate for emotion recognition system construction.

Accurate recognition and understating of human emotions is an essential skill that can improve the collaboration between humans and machines. In this vein, electroencephalogram (EEG)-based emotion recognition is considered an active research field with challenging issues regarding the analyses of the nonstationary EEG signals and the extraction of salient features that can be used to achieve accurate emotion recognition. In this paper, an EEG-based emotion recognition approach with a novel time-frequency feature extraction technique is presented. In particular, a quadratic time-frequency distribution (QTFD) is employed to construct a high resolution time-frequency representation of the EEG signals and capture the spectral variations of the EEG signals over time. To reduce the dimensionality of the constructed QTFD-based representation, a set of 13 time- and frequency-domain features is extended to the joint time-frequency-domain and employed to quantify the QTFD-based time-frequency representation of the EEG signals. Moreover, to describe different emotion classes, we have utilized the 2D arousal-valence plane to develop four emotion labeling schemes of the EEG signals, such that each emotion labeling scheme defines a set of emotion classes. The extracted time-frequency features are used to construct a set of subject-specific support vector machine classifiers to classify the EEG signals of each subject into the different emotion classes that are defined using each of the four emotion labeling schemes. The performance of the proposed approach is evaluated using a publicly available EEG dataset, namely the DEAPdataset. Moreover, we design three performance evaluation analyses, namely the channel-based analysis, feature-based analysis and neutral class exclusion analysis, to quantify the effects of utilizing different groups of EEG channels that cover various regions in the brain, reducing the dimensionality of the extracted time-frequency features and excluding the EEG signals that correspond to the neutral class, on the capability of the proposed approach to discriminate between different emotion classes. The results reported in the current study demonstrate the efficacy of the proposed QTFD-based approach in recognizing different emotion classes. In particular, the average classification accuracies obtained in differentiating between the various emotion classes defined using each of the four emotion labeling schemes are within the range of –. Moreover, the emotion classification accuracies achieved by our proposed approach are higher than the results reported in several existing state-of-the-art EEG-based emotion recognition studies.