Combination of EOG and EEG for emotion recognition over different window sizes

Emotion identification plays a vital role in human interactions. For this purpose, Computer-vision methods for automatic emotion recognition is nowadays a widely studied topic. One of the most studied approaches for automatic emotion recognition is processing multi-channel Electroencephalogram signals (EEG). This paper presents a new model for emotion recognition using brain maps as input and providing emotion states in terms of arousal and valence as output. Brain maps are a spatial representation of features extracted from EEG signals. The proposed model, called Multi-Task Convolutional Neural Network (MT-CNN), is fed with stacked brain maps of four different waves of different frequency bands: alpha, beta, gamma and theta, using differential entropy and power spectra density and considering observation windows of 0.5s. This model is trained and tested on the DEAP dataset, a well-known dataset for comparison purposes. This work shows that the MT-CNN nerforms better than other methods.

Multi-biometrics makes a big progress for the subject of biometrics. Multi-biometrics usually obtains a higher accuracy and reliability than single biometrics. Multi-biometrics depends on a fusion strategy to achieve this. The feature level and matching score level fusion seem to be two widely used and very effective fusion strategies. In this paper, we propose to combine a feature level and matching score level fusion strategies to perform personal authentication. The feature level fusion strategy fuses two biometric traits by using a PCA-based algorithm and the matching score level fusion strategy integrates the results for ultimate personal authentication. The experimental results on a multi-spectral palmprint image database show that the proposed method is feasible and effective.

The Electrooculogram (EOG) signal is often contaminated with artifacts and power-line while recording. It is very much essential to denoise the EOG signal for quality diagnosis. The present study deals with denoising of noisy EOG signals using Stationary Wavelet Transformation (SWT) technique by two different approaches, namely, increasing segments of the EOG signal and different equal segments of the EOG signal. For performing the segmental denoising analysis, an EOG signal is simulated and added with controlled noise powers of 5 dB, 10 dB, 15 dB, 20 dB, and 25 dB so as to obtain five different noisy EOG signals. The results obtained after denoising them are extremely encouraging. Root Mean Square Error (RMSE) values between reference EOG signal and EOG signals with noise powers of 5 dB, 10 dB, and 15 dB are very less when compared with 20 dB and 25 dB noise powers. The findings suggest that the SWT technique can be used to denoise the noisy EOG signal with optimum noise powers ranging from 5 dB to 15 dB. This technique might be useful in quality diagnosis of various neurological or eye disorders.

Mental disorders, especially schizophrenia, still pose a great challenge for diagnosis in early stages. Recently, computer-aided diagnosis techniques based on resting-state functional magnetic resonance imaging (Rs-fMRI) have been developed to tackle this challenge. In this work, we investigate different decision-level and feature-level fusion schemes for discriminating between schizophrenic and normal subjects. Four types of fMRI features are investigated, namely the regional homogeneity, voxel-mirrored homotopic connectivity, fractional amplitude of low-frequency fluctuations and amplitude of low-frequency fluctuations. Data denoising and preprocessing were first applied, followed by the feature extraction module. Four different feature selection algorithms were applied, and the best discriminative features were selected using the algorithm of feature selection via concave minimization (FSV). Support vector machine classifiers were trained and tested on the COBRE dataset formed of 70 schizophrenic subjects and 70 healthy subjects. The decision-level fusion method outperformed the single-feature-type approaches and achieved a 97.85% accuracy, a 98.33% sensitivity, a 96.83% specificity. Moreover, feature-fusion scheme resulted in a 98.57% accuracy, a 99.71% sensitivity, a 97.66% specificity, and an area under the ROC curve of 0.9984. In general, decision-level and feature-level fusion schemes boosted the performance of schizophrenia detectors based on fMRI features.

The aim of this study was to present electrooculogram (EOG) signals that can be used for human computer interface efficiently. Establishing an efficient alternative channel for communication without overt speech and hand movements is important to increase the quality of life for patients suffering from amyotrophic lateral sclerosis (ALS) or other diseases that prevent correct limb and facial muscular responses. Using EOG signals, it is possible to improve the communication abilities of those patients who can move their eyes. Investigating the possible usage of the EOG for human---computer interface, a relation between sight angle and EOG is determined. In other methodology, most famous approaches involve the use of a camera to visually track the eye. However, this method has problems that the eyes of user must always be open. In this paper, we propose the mouse cursor control system for ALS patients using EOG and electroencephalograph (EEG) signals. We introduced the algorithm using alternating current and direct current of EOG corresponding to the drift. Therefore, EOG measurement system we proposed improved the problems of artifacts caused by eye blinking which was not accepted for other systems, the displacement of electrode positions and the drift. In addition, we introduced the EEG measurement to examine whether the subject could control their eye movement consciously. The EEG signals were not used to control the mouse movement, but to determine the subject's control state. In order to test whether our system works well, we prepared a questionnaire and asked the subjects to operate our system, and answer with YES or NO by moving the mouse cursor. During the task, we also recorded the subjects' EEG by MYNDPLAY [7] and checked their conscious level. Three subjects participated in this experiment, and they had never operated this system before. In this experiment, we measured 30 states of EEG signals while EOG was also measuring for one eye movement and analyzed the EEG signals. The results of analysis of the EEG signal changes and the answers to questions indicated that at 26 of 30 states, the subjects' conscious level while they were moving the cursor by EOG signals was correctly determined from the EEG signals. From these results, we could know that the EEG signals can be used to adjust the EOG system whether it works according to patients' mind or just a misjudgment.

\nIn order to overcome the limitation of single mode emotion recognition. This paper describes a novel multimodal emotion recognition algorithm, and takes speech signal and facial expression signal as the research subjects. First, fuse the speech signal feature and facial expression signal feature, get sample sets by putting back sampling, and then get classifiers by BP neural network (BPNN). Second, measure the difference between two classifiers by double error difference selection strategy. Finally, get the final recognition result by the majority voting rule. Experiments show the method improves the accuracy of emotion recognition by giving full play to the advantages of decision level fusion and feature level fusion, and makes the whole fusion process close to human emotion recognition more, with a recognition rate 90.4%.\n

DEAR-MULSEMEDIA: Dataset for emotion analysis and recognition in response to multiple sensorial media

The development of suitable EEG-based emotion recognition systems has become a main target in the last decades for Brain Computer Interface applications (BCI). However, there are scarce algorithms and procedures for real-time classification of emotions. The present study aims to investigate the feasibility of real-time emotion recognition implementation by the selection of parameters such as the appropriate time window segmentation and target bandwidths and cortical regions. We recorded the EEG-neural activity of 24 participants while they were looking and listening to an audiovisual database composed of positive and negative emotional video clips. We tested 12 different temporal window sizes, 6 ranges of frequency bands and 60 electrodes located along the entire scalp. Our results showed a correct classification of 86.96% for positive stimuli. The correct classification for negative stimuli was a little bit less (80.88%). The best time window size, from the tested 1 s to 12 s segments, was 12 s. Although more studies are still needed, these preliminary results provide a reliable way to develop accurate EEG-based emotion classification.

EEG signals are the modality that is widely used to recognize human emotions. However, the limited data on EEG signals remains challenging because of the small recording participants, the need for an expert to interpret EEG signals, and the expensive cost of tools to record EEG signals. This research proposed the data augmentation schemes on the EEG datasets to overcome the limited available data problem. Augmenting the data will help the generalizability of the emotion recognition model. The EEG signals on the DEAP and SEED datasets are transformed into image samples using a recurrence plot and spectrogram. Then, the artificial recurrence plot and the artificial spectrogram samples are generated using Pix2pix. This research used these artificial samples to conduct the data augmentation process. LeNet5, ResNet50, MobileNet, and DenseNet121 are used to conduct the classification. The best four data augmentation schemes are as follows: Appending 20,000 artificial recurrence plot samples to DEAP and SEED training datasets, appending 20,000 artificial spectrogram samples to the DEAP training dataset, and appending 15,000 artificial spectrogram samples to the SEED training dataset. The kappa coefficient for each classification model based on the best data augmentation schemes is computed. It is found that among the compared classifiers, LeNet5 achieved the best accuracy in both SEED (98.58%) and DEAP (86.12%) datasets when spectrogram was used. Therefore, LeNet5 trained on the spectrogram samples is a reliable and robust classification model. This finding implies that the use of spectrogram is more promising than the recurrence plot in human emotion recognition.

Emotion recognition using EEG signals is a critical task with wide-ranging applications in mental health monitoring, disorders of consciousness, and neurofeedback systems. However, EEG signals are inherently noisy and high-dimensional, posing persistent challenges to accurate emotion classification. Most existing approaches rely on limited preprocessing or employ Principal Component Analysis (PCA) solely for dimensionality reduction, often overlooking the residual artifacts that degrade classifier performance. This study introduces a novel PCA-RF framework that repurposes PCA for dual objectives: targeted noise suppression and feature dimensionality reduction. Uniquely, PCA is applied after frequency-specific filtering to more effectively eliminate residual ocular and cardiac artifacts, thus improving the quality of EEG feature representations. These compact and denoised features are then processed by a Random Forest (RF) classifier, which robustly captures the nonlinear dynamics of the EEG data. Evaluated on the DEAP dataset, the proposed PCA-RF approach achieved state-of-the-art performance, with 96.57% accuracy for arousal and 96.03% for valence classification. The key contribution of this work lies in its strategic integration of PCA for both artifact suppression and feature optimization, setting it apart from conventional pipelines and delivering a reliable and accurate solution for EEG-based emotion recognition.

This paper proposes an efficient 1D-Binary Neural Network (1D-BNN) designed for real-time emotion recognition based on EEG signals, with a focus on optimization for deployment on wearable devices. The model processes raw EEG signals in an end-to-end manner by utilizing 1D convolution operations together with binary quantization, which enhances temporal feature extraction while significantly reducing computational complexity and memory requirements. Evaluated on the DEAP dataset, the model achieves a high classification accuracy of 97.53% for valence and 98.11% for arousal, and it reaches 98.39% for four-class emotion recognition tasks, all while maintaining an extremely compact model size of only 0.173MB. These experimental results demonstrate that the proposed 1D-BNN is highly suitable for energy-efficient, low-latency deployment in resource-constrained environments, thereby enabling practical and effective integration into wearable emotion recognition systems. Overall, this study highlights the significant potential of binarized neural networks to achieve a well-balanced trade-off between performance and computational efficiency, providing valuable insights for the development of future embedded emotion recognition applications.

In this paper, we present a comparison Support Vector Machine (SVM) and Artificial Neural Network (ANN) for classification of electrooculogram (EOG) signals acquired under specific eye movements. These methods that are required for an eye controlled system are compared by means of their accuracy and response time. Acquired EOG signals consist of 5 different eye movements - being horizontal (right and left), vertical (up and down) and blink. EOG signal acquisition was achieved from 20 different subjects by using two EOG channels (vertical and horizontal) and 3 element feature vectors were extracted. The first two elements of the feature vectors are the peak amplitudes of two channels whereas the third element, being our proposed parameter, is the kurtosis value of the active channel. 10 of 20 randomly selected feature vectors were used for training of the classifiers whereas the rest was used for performance tests. Offline tests yield 100 % success rate for both of the classifiers. The response times of both methods make them suitable for real-time usage.

Electroencephalogram (EEG) emotion recognition based on a hybrid feature extraction method in empirical mode decomposition domain combining with optimal feature selection based on sequence backward selection is proposed, which can reflect subtle information of multiscale components of unstable and nonlinear EEG signals and remove the reductant features to improve the performance of emotion recognition. The proposal is tested on DEAP dataset, in which the emotional states in the Valance dimension and Arousal dimension are classified by both ${K}$ -nearest neighbor and support vector machine, respectively. In the experiments, temporal windows of different length and three kinds of rhythms of EEG signal are taken into account for comparison, from which the results show that EEG signal with 1s temporal window achieves highest recognition accuracy of 86.46% in Valence dimension and 84.90% in Arousal dimension, respectively, which is superior to some state-of-the-art works. The proposed method would be applied to real-time emotion recognition in multimodal emotional communication-based humans–robots interaction system.

Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.

Objective: To extract statistical features from EEG signal for human emotion recognition. Methods: Emotions, the “inner” state of a person play a vital role in analysing the state of mind. In this paper, a new method for human emotion recognition using Multi Wavelet Transform (MWT) and random forest (ensemble technique) for classification of human emotions from EEG signals has been proposed. The experiment is conducted on EEG database for emotions i.e., DEAP data set. In present work, the data set used contains EEG recording from 4 participants (S01, S02, S03, S04) from 15 channel (FP1, FP2, F3, P3, F4, T7, T8, P4, O1, PZ, PO3, O2, P7, CP2, and C4) having 40 trails each. The paper explore the capability of proposed features namely Mean, Standard deviation, Variance, Shannon entropy, Hjorth parameters and Band power. Findings: The feature set obtained using Multi-Wavelet are then used as input for MLP, KNN, MC-SVM with Puk kernel function and Random Forest (ensemble) classifier for the classification of emotions. The classification accuracy for different emotion state happy 99.8%, sad 98.3%, exciting 95%, hate 96.4% are obtained by the proposed method. Overall 98.1% for classification of emotions (happy, sad, exciting, hate) from EEG signals. Application: The findings are helpful in monitoring alertness, cognitive engagement of patients with impaired motor functions; recognition of emotions; investigating sleep disorders and physiology. In future, efficient feature extraction algorithm using different multiwavelet functions and with a different set of features which include the temporal dynamics of brain signals in the human cognitive system can be considered.