Deep Learning Architectures for EEG: from CNN to Transformers

Objective. Deep transfer learning has been widely used to address the nonstationarity of electroencephalogram (EEG) data during motor imagery (MI) classification. However, previous deep learning approaches suffer from limited classification accuracy because the temporal and spatial features cannot be effectively extracted. Approach. Here, we propose a novel end-to-end deep subject adaptation convolutional neural network (SACNN) to handle the problem of EEG-based MI classification. Our proposed model jointly optimizes three modules, i.e. a feature extractor, a classifier, and a subject adapter. Specifically, the feature extractor simultaneously extracts the temporal and spatial features from the raw EEG data using a parallel multiscale convolution network. In addition, we design a subject adapter to reduce the feature distribution shift between the source and target subjects by using the maximum mean discrepancy. By minimizing the classification loss and the distribution discrepancy, the model is able to extract the temporal-spatial features to the prediction of a new subject. Main results. Extensive experiments are carried out on three EEG-based MI datasets, i.e. brain–computer interface (BCI) competition IV dataset IIb, BCI competition III dataset IVa, and BCI competition IV dataset I, and the average accuracy reaches to 86.42%, 81.71% and 79.35% on the three datasets respectively. Furthermore, the statistical analysis also indicates the significant performance improvement of SACNN. Significance. This paper reveals the importance of the temporal-spatial features on EEG-based MI classification task. Our proposed SACNN model can make fully use of the temporal-spatial information to achieve the purpose.

Electroencephalography (EEG) motor imagery signals have recently gained significant attention due to its ability to encode a person’s intent to perform an action. Researchers have used motor imagery signals to help disabled persons control devices, such as wheelchairs and even autonomous vehicles. Hence, the accurate decoding of these signals is important to brain–computer interface (BCI) systems. Such motor imagery-based BCI systems can become an integral part of cognitive modules that are increasingly being used in smart city frameworks. However, the classification and recognition of EEG have consistently been a challenge due to its dynamic time series data and low signal-to-noise ratio. Deep learning methods, such as the convolution neural network (CNN), have achieved remarkable success in computer vision tasks. Considering the limited applications of deep learning for motor imagery EEG classification, this work focuses on developing CNN-based deep learning methods for such purpose. We propose a multiple-CNN feature fusion architecture to extract and fuse features by using subject-specific frequency bands. CNN has been designed with variable filter sizes and split convolutions for the extraction of spatial and temporal information from raw EEG data. A feature fusion technique based on autoencoders is applied. Cross-encoding technique has been proposed and is successfully used to train autoencoders for a novel cross-subject information transfer and augmenting EEG data. This proposed method outperforms the state-of-the-art four-class motor imagery classification methods for subject-specific and cross-subject data. Autoencoder cross-encoding helps to learn subject invariant and generic features for EEG data and achieves more than 10% increase on cross-subject classification results. The fusion approaches show the potential of applying multiple CNN feature fusion techniques for the advancement of EEG-related research.

Automatic high-level feature extraction has become a possibility with the advancement of deep learning, and it has been used to optimize efficiency. Recently, classification methods for Convolutional Neural Network (CNN)-based electroencephalography (EEG) motor imagery have been proposed, and have achieved reasonably high classification accuracy. These approaches, however, use the CNN single convolution scale, whereas the best convolution scale varies from subject to subject. This limits the precision of classification. This paper proposes multibranch CNN models to address this issue by effectively extracting the spatial and temporal features from raw EEG data, where the branches correspond to different filter kernel sizes. The proposed method’s promising performance is demonstrated by experimental results on two public datasets, the BCI Competition IV 2a dataset and the High Gamma Dataset (HGD). The results of the technique show a 9.61% improvement in the classification accuracy of multibranch EEGNet (MBEEGNet) from the fixed one-branch EEGNet model, and 2.95% from the variable EEGNet model. In addition, the multibranch ShallowConvNet (MBShallowConvNet) improved the accuracy of a single-scale network by 6.84%. The proposed models outperformed other state-of-the-art EEG motor imagery classification methods.

Objective. Noninvasive brain–computer interfaces (BCIs) assist paralyzed patients by providing access to the world without requiring surgical intervention. Prior work has suggested that EEG motor imagery based BCI can benefit from increased decoding accuracy through the application of deep learning methods, such as convolutional neural networks (CNNs). Approach. Here, we examine whether these improvements can generalize to practical scenarios such as continuous control tasks (as opposed to prior work reporting one classification per trial), whether valuable information remains latent outside of the motor cortex (as no prior work has compared full scalp coverage to motor only electrode montages), and the existing challenges to the practical implementation of deep-learning based continuous BCI control. Main results. We report that: (1) deep learning methods significantly increase offline performance compared to standard methods on an independent, large, and longitudinal online motor imagery BCI dataset with up to 4-classes and continuous 2D feedback; (2) our results suggest that a variety of neural biomarkers for BCI, including those outside the motor cortex, can be detected and used to improve performance through deep learning methods, and (3) tuning neural network output will be an important step in optimizing online BCI control, as we found the CNN models trained with full scalp EEG also significantly reduce the average trial length in a simulated online cursor control environment. Significance. This work demonstrates the benefits of CNNs classification during BCI control while providing evidence that electrode montage selection and the mapping of CNN output to device control will be important design choices in CNN based BCIs.

Abstract—Alzheimer disease (AD) is a progressive neurodegen- erative disorder, treatment and management of which needs to be timely. Conventional diagnosis techniques such as MRI, PET, and cerebrospinal fluid test are mostly invasive, costly, or not available at large scale. On the other hand, electroencephalography (EEG) represents a non-invasive, low-cost, and portable method of brain monitoring, hence, making it a favorable modality to use to screen AD at an early onset. The article contains a deep neural network framework of automatic EEG-based Alzheimer disease classification. Raw EEG data were processed and transformed to time-frequency representation and connection matrices. The different architectures (convolutional neural networks (CNNs), hybrid CNN-LSTM models as well as transformer-based models) were trained and compared with the conventional machine learning-based models (support vector machines and random forests). The deep neural models were found to be more effective than the traditional methods with improved accuracy, sensitiv- ity and specificity in distinguishing between the patients with AD and mild cognitive impairment (MCI) and those without mental illnesses. On top of highlighting frequency bands and areas that are correlated with known AD biomarkers, which is important to validate the clinical relevance of the finding, model interpretability methods, such as Grad-CAM, also focused on frequency bands and areas that are correlated with known AD biomarkers. These findings indicate that EEG-based deep neural models can be used practically to scalably and early diagnose the presence of Alzheimer as not only deep learning using EEG signals improves diagnostic accuracy but also provides biologically relevant outcomes. Index Terms—Alzheimer’s disease, Electroencephalography (EEG), Deep learning, Neural networks, Convolutional neural networks (CNN), Long short-term memory (LSTM), Trans- formers, Mild cognitive impairment (MCI), Early diagnosis, Brain–computer interface (BCI), Medical signal processing.

Recent advances in deep learning have had a methodological and practical impact on brain-computer interface (BCI) research. Among the various deep network architectures, convolutional neural networks (CNNs) have been well suited for spatio-spectral-temporal electroencephalogram (EEG) signal representation learning. Most of the existing CNN-based methods described in the literature extract features at a sequential level of abstraction with repetitive nonlinear operations and involve densely connected layers for classification. However, studies in neurophysiology have revealed that EEG signals carry information in different ranges of frequency components. To better reflect these multi-frequency properties in EEGs, we propose a novel deep multi-scale neural network that discovers feature representations in multiple frequency/time ranges and extracts relationships among electrodes, i.e., spatial representations, for subject intention/condition identification. Furthermore, by completely representing EEG signals with spatio-spectral-temporal information, the proposed method can be utilized for diverse paradigms in both active and passive BCIs, contrary to existing methods that are primarily focused on single-paradigm BCIs. To demonstrate the validity of our proposed method, we conducted experiments on various paradigms of active/passive BCI datasets. Our experimental results demonstrated that the proposed method achieved performance improvements when judged against comparable state-of-the-art methods. Additionally, we analyzed the proposed method using different techniques, such as PSD curves and relevance score inspection to validate the multi-scale EEG signal information capturing ability, activation pattern maps for investigating the learned spatial filters, and t-SNE plotting for visualizing represented features. Finally, we also demonstrated our method's application to real-world problems. Based on our experimental results and analyses, we believe that the proposed multi-scale neural network can be useful for various BCI paradigms, as a starting model or as a backbone network in any new BCI experiments.

Brain–computer interface (BCI) based on motor imagery (MI) supplies a communication method between humans and machines independent of external stimulation. MI is based on the imagination quality of the subjects, which resulted in large variability and unreliability of the signals. Improved MI paradigms and data analysis methods would be helpful to reduce the side effects of the individual specificity and temporal variation, and increase the classification performance of MI-based BCI. Visual guidance could improve the MI quality, but there is a lack of a systematic analysis of the forms of visual guidance and their effects on MI. In this article, we conducted a comparative study to evaluate the MI paradigms with different visual guidance manners. Five paradigms were designed, i.e., pure MI, MI with synchronous pursuit (SP), asynchronous pursuit (AP), synchronous saccade (SS), and asynchronous saccade (AS). Furthermore, a convolutional neural network (CNN) architecture with multiple receptive fields and attention module was proposed. Twenty subjects accomplished the experiments. The SP paradigm induced the most significant event-related desynchronization (ERD) phenomenon and the highest classification accuracy for electroencephalograph (EEG) signals. The proposed network achieved an average classification accuracy of 91.89% and standard deviation of 5.55%, which outperformed the compared methods. To test the applicability of the paradigm and the method, six subjects with different performance in the offline experiment then participated in an online experiment and a simulated brain-controlled trajectory tracking flight of the unmanned aerial vehicle (UAV). All the subjects could accomplish the task, and their performance was positively correlated with the classification accuracy and negatively correlated with the complexity of the tracking path. In general, the SP visual guidance effectively helped the subjects modulate their brain activity, leading to increased MI quality. The convolutional network with multiple receptive fields and an attention module showed the improved classification performance and robustness.

Recent advancements in deep learning techniques tailored for autonomous driving in Indian road conditions are crucial for revolutionizing transportation systems. Indian roads present unique challenges, including unpredictable traffic patterns, diverse road infrastructures, and challenging weather conditions. Deep learning is pivotal in addressing these challenges, focusing on perception, decision-making, and control. Various architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Deep Reinforcement Learning (DRL), are analyzed for their efficacy in autonomous driving. Sensor fusion techniques enhance perception capabilities, such as lidar, radar, and camera data integration. Additionally, advancements in semantic segmentation and object detection algorithms improve scene understanding and obstacle recognition. Collaborative efforts between academia, industry, and government agencies are essential for accelerating the deployment of autonomous driving technology in Indian road conditions. In conclusion, leveraging cutting-edge deep learning methodologies enables safe and efficient navigation of autonomous vehicles, ushering in a transformative future in transportation systems.

Advancements in neuroscience and deep learning have significantly enhanced bio-signal-based emotion recognition, a critical component in Brain-Machine Interface (BMI) applications for healthcare, human-computer interaction, and human-AI assistant communication. Former studies have proposed Manual Mapping electrode matrices and employing Convolutional Neural Networks (CNNs) to recognize spatial EEG activities. However, this Manual Mapping of EEG electrodes onto matrix grids limits spatial precision and introduces inefficiencies. This study proposes automated channel mapping methods of Orthographic Projection and Stereographic Projection to address these challenges, using Differential Entropy and Power Spectral Density with Linear Dynamical Systems as features. A 3-branch multiscale CNN was trained on open-source dataset, employing a 5-fold cross-classification approach. Experimental results demonstrate that higher-resolution grids (16×16, 24×24) with automated projections significantly outperform Manual Mappings, achieving up to a 4.06% improvement in classification accuracy (p < 0.05). This result indicates that enhancing spatial precision of EEG data improves emotion classification, establishing automated spatial mapping as an advancement in EEG-based emotion recognition.Clinical Relevance-Advancement in emotion classification accuracy can facilitate more reliable diagnostic tools and personalized therapeutic interventions for mental health disorders, such as depression and anxiety.

Motor imagery (MI) is a cognitive process wherein an individual mentally rehearses a specific movement without physically executing it. Recently, MI-based brain-computer interface (BCI) has attracted widespread attention. However, accurate decoding of MI and understanding of neural mechanisms still face huge challenges. These seriously hinder the clinical application and development of BCI systems based on MI. Thus, it is very necessary to develop new methods to decode MI tasks. In this work, we propose a multi-branch convolutional neural network (MBCNN) with a temporal convolutional network (TCN), an end-to-end deep learning framework to decode multi-class MI tasks. We first used MBCNN to capture the MI electroencephalography signals information on temporal and spectral domains through different convolutional kernels. Then, we introduce TCN to extract more discriminative features. The within-subject cross-session strategy is used to validate the classification performance on the dataset of BCI Competition IV-2a. The results showed that we achieved 75.08% average accuracy for 4-class MI task classification, outperforming several state-of-the-art approaches. The proposed MBCNN-TCN-Net framework successfully captures discriminative features and decodes MI tasks effectively, improving the performance of MI-BCIs. Our findings could provide significant potential for improving the clinical application and development of MI-based BCI systems.

Emotion detection, hence, is an important step toward making human-computer interaction a more enhanced process, where systems are made capable of identifying and responding to the emotional state of users. In fact, multimodal emotion detection systems in which both auditory and visual information are fused are emerging, and these approaches toward expressive emotional states are complementary and robust. Multimodal systems enhance the quality of interacting and, through many applications, can diagnose emotional disorders, monitor automotive safety, and improve human-robot interactions. In nature, the high-dimensional space and dynamic threats have resulted in obtaining low accuracy and high computational cost in applying the traditional models based on single-modality data. On the other hand, multimodal systems explore the synergy between audio and visual data, giving better performance and higher accuracy in inferring subtle emotional expressions. The latest improvement was done on these systems using recent advancements in transfer learning and deep learning techniques.That being said, this research Proposal devises a multimodal emotion recognition system integrating speech and face information through transfer learning for improved accuracy and robustness. Serving this purpose, the objectives of this research entail the effective comparison among different transfer-learning strategies, including the impact of pre-trained models in speech-based emotion recognition, and to introduce the role of voice activity detection in the process. Advanced neural network architectures like Spatial Transformer Networks and bidirectional LSTM in facial emotion recognition will also be tested. Early and late fusion strategies will also be used to find the best strategy for combining speech and facial data.This research will target several challenges that involve the complexity of data, balancing of the model performance-robustness balance, computational limitations, and standardization of evaluations in developing a working and robust emotion recognition system to enhance digital interaction and apply in practical areas. The goal is to create a system that oversteps the limitation of single-modality models through state-of-the-art advances in deep learning, as well as front-line improvements in transfer learning, in the manner of emotion detection performance.

Following recent advancements in deep learning and artificial intelligence, spoken language identification applications are playing an increasingly significant role in our day-to-day lives, especially in the domain of multi-lingual speech recognition. In this article, we propose a spoken language identification system that depends on the sequence of feature vectors. The proposed system uses a hybrid Convolutional Recurrent Neural Network (CRNN), which combines a Convolutional Neural Network (CNN) with a Recurrent Neural Network (RNN) network, for spoken language identification on seven languages, including Arabic, chosen from subsets of the Mozilla Common Voice (MCV) corpus. The proposed system exploits the advantages of both CNN and RNN architectures to construct the CRNN architecture. At the feature extraction stage, it compares the Gammatone Cepstral Coefficient (GTCC) feature and Mel Frequency Cepstral Coefficient (MFCC) feature, as well as a combination of both. Finally, the speech signals were represented as frames and used as the input for the CRNN architecture. After conducting experiments, the results of the proposed system indicate higher performance with combined GTCC and MFCC features compared to GTCC or MFCC features used individually. The average accuracy of the proposed system was 92.81% in the best experiment for spoken language identification. Furthermore, the system can learn language-specific patterns in various filter size representations of speech files.

BackgroundThe aging of the world population poses a major health challenge, and brain–computer interface (BCI) technology has the potential to provide assistance and rehabilitation for the elderly.ObjectivesThis study aimed to investigate the electroencephalogram (EEG) characteristics during motor imagery by comparing young and elderly, and study Convolutional Neural Networks (CNNs) classification for the elderly population in terms of fatigue analysis in both frontal and parietal regions.MethodsA total of 20 healthy individuals participated in the study, including 10 young and 10 older adults. All participants completed the left- and right-hand motor imagery experiment. The energy changes in the motor imagery process were analyzed using time–frequency graphs and quantified event-related desynchronization (ERD) values. The fatigue level of the motor imagery was assessed by two indicators: (θ + α)/β and θ/β, and fatigue-sensitive channels were distinguished from the parietal region of the brain. Then, rhythm entropy was introduced to analyze the complexity of the cognitive activity. The phase-lock values related to the parietal and frontal lobes were calculated, and their temporal synchronization was discussed. Finally, the motor imagery EEG data was classified by CNNs, and the accuracy was discussed based on the analysis results.ResultFor the young and elderly, ERD was observed in C3 and C4 channels, and their fatigue-sensitive channels in the parietal region were slightly different. During the experiment, the rhythm entropy of the frontal lobe showed a decreasing trend with time for most of the young subjects, while there was an increasing trend for most of the older ones. Using the CNN classification method, the elderly achieved around 70% of the average classification accuracy, which is almost the same for the young adults.ConclusionCompared with the young adults, the elderly are less affected by the level of cognitive fatigue during motor imagery, but the classification accuracy of motor imagery data in the elderly may be slightly lower than that in young persons. At the same time, the deep learning method also provides a potentially feasible option for the application of motor-imagery BCI (MI-BCI) in the elderly by considering the ERD and fatigue phenomenon together.

Recent advancements in deep learning, particularly convolutional networks, have rapidly become the preferred methodology for analyzing medical images, facilitating tasks like disease segmentation, classification, and pattern quantification. Central to these advancements is the capacity to leverage hierarchical feature representations acquired solely from data. This comprehensive review meticulously examines a variety of deep learning techniques applied across diverse healthcare domains, delving into the intricate realm of medical imaging to unveil concealed patterns through strategic deep learning methodologies. Encompassing a range of diseases, including Alzheimer’s, breast cancer, brain tumors, glaucoma, heart murmurs, retinal microaneurysms, colorectal liver metastases, and more, the analysis emphasizes contributions succinctly summarized in a tabular form.The table provides an overview of various deep learning approaches applied to different diseases, incorporating methodologies, datasets, and outcomes for each condition.Notably, performance metrics such as accuracy, specificity, sensitivity, and other crucial measures underscore the achieved results. Specifically, an in-depth discussion is conducted on the Convolutional Neural Network (CNN) owing to its widespread adoption as a paramount tool in computer vision tasks. Moreover, an exhaustive exploration encompasses deep learning classification approaches, procedural aspects of medical image processing, as well as a thorough examination of key features and characteristics. At the end, we delve into a range of research challenges and put forth potential avenues for future improvements in the field.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.