Dataset-Independent EEG Channel Selection for Emotion Recognition.

In our life, emotions often have a profound impact on human behavior, especially for drivers, as negative emotions can increase the risk of traffic accidents. As such, it is imperative to accurately discern the emotional states of drivers in order to preemptively address and mitigate any negative emotions that may otherwise manifest and compromise driving behavior. In contrast to many current studies that rely on complex and deep neural network models to achieve high accuracy, this research aims to explore the potential of achieving high recognition accuracy using shallow neural networks through restructuring the structure and dimensions of the data. In this study, we propose an end-to-end convolutional neural network (CNN) model called simply ameliorated CNN (SACNN) to address the issue of low accuracy in cross-subject emotion recognition. We extracted features and converted dimensions of EEG signals from the SEED dataset from the BCMI Laboratory to construct 62-dimensional data, and obtained the optimal model configuration through ablation experiments. To further improve recognition accuracy, we selected the top 10 channels with the highest accuracy by separately training the EEG data of each of the 62 channels. The results showed that the SACNN model achieved an accuracy of 88.16% based on raw cross-subject data, and an accuracy of 91.85% based on EEG channel data from the top 10 channels. In addition, we explored the impact of the position of the BN and dropout layers on the model through experiments, and found that a targeted shallow CNN model performed better than deeper and larger perceptual field CNN models. Furthermore, we discuss herein the future issues and challenges of driver emotion recognition in promising smart city applications.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

During electroencephalography (EEG)-based motor imagery-brain-computer interfaces (MI-BCIs) task, a large number of electrodes are commonly used, and consume much computational resources. Therefore, channel selection is crucial while ensuring classification accuracy. This paper proposes a channel selection method by integrating the efficient channel attention (ECA) module with a convolutional neural network (CNN). During model training process, the ECA module automatically assigns the channel weights by evaluating the relative importance for BCI classification accuracy of every channel. Then a ranking of EEG channel importance can be established so as to select an appropriate number of channels to form a channel subset from the ranking. In this paper, the ECA module is embedded into a commonly used network for MI, and comparative experiments are conducted on the BCI Competition IV dataset 2a. The proposed method achieved an average accuracy of 75.76% with all 22 channels and 69.52% with eight channels in a four-class classification task, outperforming other state-of-the-art EEG channel selection methods. The result demonstrates that the proposed method provides an effective channel selection approach for EEG-based MI-BCI.

Accurately identifying epileptogenic zone (EZ) using high-frequency oscillations (HFOs) is a challenge that must be mastered to transfer HFOs into clinical use. We analyzed the ability of a convolutional neural network (CNN) model to distinguish EZ and non-EZ HFOs. Nineteen medically intractable epilepsy patients with good surgical outcomes 2 years after surgery were studied. Five-minute interictal intracranial electroencephalogram epochs of slow-wave sleep were selected randomly. Then 5 s segments of ripples (80–200 Hz) and fast ripples (FRs, 200–500 Hz) were detected automatically. The EZs and non-EZs were identified using the surgery resection range. We innovatively converted all epochs into four types of images using two scales: original waveforms, filtered waveforms, wavelet spectrum images, and smoothed pseudo Wigner–Ville distribution (SPWVD) spectrum images. Two scales were fixed and fitted scales. We then used a CNN model to classify the HFOs into EZ and non-EZ categories. As a result, 7,000 epochs of ripples and 2,000 epochs of FRs were randomly selected from the EZ and non-EZ data for analysis. Our CNN model can distinguish EZ and non-EZ HFOs successfully. Except for original ripple waveforms, the results from CNN models that are trained using fixed-scale images are significantly better than those from models trained using fitted-scale images (p < 0.05). Of the four fixed-scale transformations, the CNN based on the adjusted SPWVD (ASPWVD) produced the best accuracies (80.89 ± 1.43% and 77.85 ± 1.61% for ripples and FRs, respectively, p < 0.05). The CNN using ASPWVD transformation images is an effective deep learning method that can be used to classify EZ and non-EZ HFOs.

Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals

In order to realize high-accuracy recognition of aerobics actions, a highly applicable deep learning model and faster data processing methods are required. Therefore, it is a major difficulty in the field of research on aerobics action recognition. Based on this, this paper studies the application of the convolution neural network (CNN) model combined with the pyramid algorithm in aerobics action recognition. Firstly, the basic architecture of the convolution neural network model based on the pyramid algorithm is proposed. Combined with the application strategy of the common recognition model in aerobics action recognition, the traditional aerobics action capture information is processed. Through the characteristics of different aerobics actions, different accurate recognition is realized, and then, the error of the recognition model is evaluated. Secondly, the composite recognition function of the convolution neural network model in this application is constructed, and the common data layer effect recognition method is used in the optimization recognition. Aiming at the shortcomings of the composite recognition function, the pyramid algorithm is used to improve the convolution neural network recognition model by deep learning optimization. Finally, through the effectiveness comparison experiment, the results show that the convolution neural network model based on the pyramid algorithm is more efficient than the conventional recognition method in aerobics action recognition.

Among the many deep learning methods, the convolutional neural network (CNN) model has an excellent performance in image recognition. Research on identifying and classifying image datasets using CNN is ongoing. Animal species recognition and classification with CNN is expected to be helpful for various applications. However, sophisticated feature recognition is essential to classify quasi-species with similar features, such as the quasi-species of parrots that have a high color similarity. The purpose of this study is to develop a vision-based mobile application to classify endangered parrot species using an advanced CNN model based on transfer learning (some parrots have quite similar colors and shapes). We acquired the images in two ways: collecting them directly from the Seoul Grand Park Zoo and crawling them using the Google search. Subsequently, we have built advanced CNN models with transfer learning and trained them using the data. Next, we converted one of the fully trained models into a file for execution on mobile devices and created the Android package files. The accuracy was measured for each of the eight CNN models. The overall accuracy for the camera of the mobile device was 94.125%. For certain species, the accuracy of recognition was 100%, with the required time of only 455 ms. Our approach helps to recognize the species in real time using the camera of the mobile device. Applications will be helpful for the prevention of smuggling of endangered species in the customs clearance area.

Emotion recognition (ER) is crucial for understanding human behaviours, social interactions, and psychological well-being. Electroencephalography (EEG) has emerged as a promising tool for capturing the neural correlates of emotions. This work is a systematic review of articles in ER using EEG signals. A total of 120 articles from 1041 articles were selected based on PRISMA guidelines using defined inclusion and exclusion criteria, published between 2018 and 2024. This article aims to provide an in-depth understanding of the current landscape of ER from EEG signals utilizing deep learning (DL). This review offers valuable guidance for researchers and practitioners seeking more refined and reliable emotion classification systems. To explore the effectiveness of DL models in EEG-based ER, several potential DL models, such as convolutional neural network, long short-term memory (LSTM), gated recurrent unit (GRU), hybrid bidirectional LSTM (BiLSTM), bidirectional GRU, and advanced DL models such as convolutional recurrent neural network and EEG-Conformer models are applied to two popular datasets, SEED and GAMEEMO, respectively, to depict the full process of ER. Additionally, the performance of DL models is also compared with the performance of basic machine learning (ML) models such as SVM, k-nearest neighbors, logistic regression, and boosting algorithms such as AdaBoost, XGBoost and LightGBM. Through extensive experiments and performance evaluations, the performance of different models when applied to the datasets mentioned above is compared. The accuracy, precision, recall, and F1-scores are analysed to determine the most effective model for EEG-based ER. The findings of this study demonstrate that the performance of hybrid DL models is more efficacious than that of ML models. The best-performing model (BiLSTM) classified the emotions, with an accuracy of 90.54% when applied to the GAMEEMO dataset. This research contributes to the growing body of literature on ER and provides insights into the feasibility of using EEG signals to understand emotional states, and presents a structured roadmap for future exploration. The findings can aid in the development of more accurate and reliable ER systems, which can have wide-ranging applications in psychology, social sciences, and human-computer interactions.

Emotions expressed by humans can be identified from facial expressions, speech signals, or physiological signals. Among them, the use of physiological signals for emotion classification is a notable emerging area of research. In emotion recognition, a person’s electrocardiogram (ECG) and galvanic skin response (GSR) signals cannot be manipulated, unlike facial and voice signals. Moreover, wearables such as smartwatches and wristbands enable the detection of emotions in people’s naturalistic environment. During the COVID-19 pandemic, it was necessary to detect people’s emotions in order to ensure that appropriate actions were taken according to the prevailing situation and achieve societal balance. Experimentally, the duration of the emotion stimulus period and the social and non-social contexts of participants influence the emotion classification process. Hence, classification of emotions when participants are exposed to the elicitation process for a longer duration and taking into consideration the social context needs to be explored. This work explores the classification of emotions using five pretrained convolutional neural network (CNN) models: MobileNet, NASNetMobile, DenseNet 201, InceptionResnetV2, and EfficientNetB7. The continuous wavelet transform (CWT) coefficients were detected from ECG and GSR recordings from the AMIGOS database with suitable filtering. Scalograms of the sum of frequency coefficients versus time were obtained and converted into images. Emotions were classified using the pre-trained CNN models. The valence and arousal emotion classification accuracy obtained using ECG and GSR data were, respectively, 91.27% and 91.45% using the InceptionResnetV2 CNN classifier and 99.19% and 98.39% using the MobileNet CNN classifier. Other studies have not explored the use of scalograms to represent ECG and GSR CWT features for emotion classification using deep learning models. Additionally, this study provides a novel classification of emotions built on individual and group settings using ECG data. When the participants watched long-duration emotion elicitation videos individually and in groups, the accuracy was around 99.8%. MobileNet had the highest accuracy and shortest execution time. These subject-independent classification methods enable emotion classification independent of varying human behavior.

Purpose: Parkinson’s disease (PD) diagnosis algorithms based on Quantitative Susceptibility Mapping and image algorithms rely on SN labeling. However, the difference between SN labels from different experts (or segmentation algorithms) will have a negative impact on downstream diagnostic tasks, such as the decrease of the algorithm's accuracy or different diagnostic results for the same sample. In this article, we quantify the accuracy of the algorithm on different label sets, and then improve the Convolutional Neural Network Model to obtain a high-precision and high-robust diagnosis algorithm. Methods: The logistic regression model and convolutional neural network (CNN) model were first compared for classification between PD patients and healthy controls (HC), given different sets of substantia nigra (SN) labeling. Then, based on the CNN model with better performance, we further proposed a novel "Gated Pooling" operation and integrated it with deep learning to attain a joint framework for image segmentation and classification. Results: The experimental results show that, with different sets of SN labeling that mimic different experts, the CNN model can maintain the stable classification accuracy around 86.4%, while the conventional logistic regression model yields a large fluctuation ranging from 78.9% to 67.9%. Furthermore, the "Gated Pooling" operation after being integrated for joint image segmentation and classification can improve the diagnosis accuracy to 86.9% consistently, which is statistically better than the baseline. Conclusion: The CNN model, compared with the conventional logistic regression model using radiomics features, has better stability in PD diagnosis. Furthermore, the joint end-to-end CNN model is shown to be suitable for PD diagnosis from the perspectives of accuracy, stability, and convenience in actual use.

Understanding the learning mechanism of convolutional neural networks in spectral analysis

The use of neural networks to solve video analytics problems in the agro-industrial complex requires training models on specialized data sets. Preparing a data set for training a neural network is the most time-consuming part of training and in some cases determines the feasibility of using artificial intelligence to solve the task. Given the growing need of agro-industrial enterprises for specialized models, the study of methods to improve the efficiency of data markup for training neural networks seems relevant. (Research purpose) The research purpose is developing a method of automatic data markup for training a convolutional neural network. (Materials and methods) YOLOv5 convolutional neural network were used. The following software was used: CMS (Content Management System), Bandicam, Hand Brake, Sony Vegas 12, Search My Files, Visual Studiocode; programming was conducted in Python. (Results and discussion) It were determined that the basic models of the YOLOv5 convolutional neural network have different levels of object recognition probability. It was shown that under the conditions of the experiment, when detecting through the cell lattice, the most voluminous model detected an object in the image at 68.2 percent of frames, while the correct detection was noted at 0.9 percent of frames with detection. It was found that with manual marking of the training data set, the average marking speed is 5.2 ± 0.34 seconds per frame containing one object, the time spent on manual marking is up to 95 percent of the total time spent on preparing the training data set. The presented algorithm of actions was used to exclude the stage of manual data markup. (Conclusions) It was found that the use of existing basic models of convolutional neural networks at the stage of preparing a data set allows minimizing labor costs at the manual markup phase. It was stated that the determination of the cost ratio in the training and use of specialized and general models of neural networks requires additional research.

Text sentiment analysis is used to discover the public’s appreciation and preferences for specific events. In order to effectively extract the deep semantic features of sentences and reduce the dependence of long distance information dependency, two models based on convolutional neural network and bidirectional long short-term memory model, CNN-BLSTM and BLSTM-CNN are proposed. Convolutional neural networks can get text features better. Bidirectional long-short time memory model can not only capture long-range information and solve gradient attenuation problem, but also represent future contextual information semantics of word sequence better. These two network architectures are explained in detail in this paper, and performed comparisons against some normal methods, such as methods based emotion lexicon, machine learning methods, LSTM and other neural network models. Experiments show that these two proposed models have achieved better results in text sentiment analysis. The best model CNN-BLSTM is better than the normal neural network models in accuracy.

This study compares the 5-day tropical cyclone (TC) track forecasts from the Global Ensemble Forecast System (GEFS) with a machine learning approach applied to this ensemble. A convolutional neural network (CNN) model was developed using northern Atlantic TC training data from 2008 to 2022, while an independent set of storms was used to evaluate the CNN model. The CNN for tropical cyclone track forecasting was trained using fivefold cross validation on a shuffled dataset split into 90% training/validation and 10% evaluation. PyTorch was used to develop a CNN model with a custom loss function, and performance was assessed using the haversine function and error metrics, comparing the CNN’s TC track forecasts to GEFS mean track forecasts. The CNN has a 58%–86% better track prediction against GEFS mean for forecast hours 0–96, which decreases to 35%–53% for hours 108–120. Track differences between those cases that the CNN improved the forecast versus did not improve are also explored, which shows that most of the CNN improvement is from a decrease in the along-track error (ATE). This is consistent with past studies of this ensemble, which showed that the largest bias exists in the along-track direction. Significance Statement This research focuses on enhancing the accuracy of tropical cyclone track forecasts by leveraging a convolutional neural network (CNN). Traditional forecasting models have relatively large errors in predicting the track of tropical cyclones, especially at forecast lead times of 3–5 days. These errors can result in inadequate warnings and misallocation of resources during storm preparations, potentially endangering lives and property. Our CNN model addresses these issues by correcting systematic biases, resulting in more accurate forecasts.

Detection of air traffic conflicts in a weather constrained airspace is challenging given the inherent uncertainties and aircraft maneuvers which give rise to new conflict birth-points. Traditional conflict detection tools are untenable in such situations as they primarily rely on flight-plan, aircraft performance characteristics and trajectories projection in short-term (2-4 minutes). This work adopts a convolutional neural network (CNN) model, on radar-like images, for conflict detection task in a constrained airspace. The CNN models are well-known for their learning capabilities when dealing with unstructured data like pixelated images. In this study, historical ADS-B data with weather constrained airspace is input as pixelated images to the CNN model. The learned model was compared with two well-known models for conflict detection (CD). The results demonstrated that the CNN based model was able to predict off-nominal conflict with high accuracy. The CNN model also demonstrated its ability to predict off-nominal conflict early for a given ten-minute look-ahead window. The CNN based model also showed low levels of false alarm signals as compared to other models. Generally speaking, all models showed low probabilities of miss-detection, mostly in the early phase of the 10-minute look-ahead window. This novel approach may serve to develop effective CD algorithms with longer look-ahead time and may aid in early detection of air traffic conflicts in non-nominal scenarios.