A novel Morse code-inspired method for multiclass motor imagery brain–computer interface (BCI) design

Objective. We proposed a novel simultaneous hybrid brain–computer interface (BCI) by incorporating electrical stimulation into a motor imagery (MI) based BCI system. The goal of this study was to enhance the overall performance of an MI-based BCI. In addition, the brain oscillatory pattern in the hybrid task was also investigated. Approach. 64-channel electroencephalographic (EEG) data were recorded during MI, selective attention (SA) and hybrid tasks in fourteen healthy subjects. In the hybrid task, subjects performed MI with electrical stimulation which was applied to bilateral median nerve on wrists simultaneously. Main results. The hybrid task clearly presented additional steady-state somatosensory evoked potential (SSSEP) induced by electrical stimulation with MI-induced event-related desynchronization (ERD). By combining ERD and SSSEP features, the performance in the hybrid task was significantly better than in both MI and SA tasks, achieving a ~14% improvement in total relative to the MI task alone and reaching ~89% in mean classification accuracy. On the contrary, there was no significant enhancement obtained in performance while separate ERD feature was utilized in the hybrid task. In terms of the hybrid task, the performance using combined feature was significantly better than using separate ERD or SSSEP feature. Significance. The results in this work validate the feasibility of our proposed approach to form a novel MI-SSSEP hybrid BCI outperforming a conventional MI-based BCI through combing MI with electrical stimulation.

Motor imagery (MI) is an important brain-computer interface (BCI) paradigm. The traditional MI paradigm (imagining different limbs) limits the intuitive control of the outer devices, while fine MI paradigm (imagining different joint movements from the same limb) can control the mechanical arm without cognitive disconnection. However, the decoding performance of fine MI limits its application. Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are widely used in BCI systems because of their portability and easy operation. In this study, a fine MI paradigm including four classes (hand, wrist, shoulder and rest) was designed, and the data of EEG-fNIRS bimodal brain activity was collected from 12 subjects. Event-related desynchronization (ERD) from EEG signals shows a contralateral dominant phenomenon, and there is difference between the ERD of the four classes. For fNIRS signal in the time dimension, the time periods with significant difference can be observed in the activation patterns of four MI tasks. Spatially, the signal peak based brain topographic map also shows difference of these four MI tasks. The EEG signal and fNIRS signal of these four classes are distinguishable. In this study, a bimodal fusion network is proposed to improve the fine MI tasks decoding performance. The features of these two modalities are extracted separately by two feature extractors based on convolutional neural networks (CNN). The recognition performance was significantly improved by the bimodal method proposed in this study, compared with the performance of the single-modal network. The proposed method outperformed all comparison methods, and achieved a four-class accuracy of 58.96%. This paper demonstrates the feasibility of EEG and fNIRS bimodal BCI systems for fine MI, and shows the effectiveness of the proposed bimodal fusion method. This research is supposed to support fine MI-based BCI systems with theories and techniques.

Motor imagery (MI) is a cognitive process wherein an individual mentally rehearses a specific movement without physically executing it. Recently, MI-based brain-computer interface (BCI) has attracted widespread attention. However, accurate decoding of MI and understanding of neural mechanisms still face huge challenges. These seriously hinder the clinical application and development of BCI systems based on MI. Thus, it is very necessary to develop new methods to decode MI tasks. In this work, we propose a multi-branch convolutional neural network (MBCNN) with a temporal convolutional network (TCN), an end-to-end deep learning framework to decode multi-class MI tasks. We first used MBCNN to capture the MI electroencephalography signals information on temporal and spectral domains through different convolutional kernels. Then, we introduce TCN to extract more discriminative features. The within-subject cross-session strategy is used to validate the classification performance on the dataset of BCI Competition IV-2a. The results showed that we achieved 75.08% average accuracy for 4-class MI task classification, outperforming several state-of-the-art approaches. The proposed MBCNN-TCN-Net framework successfully captures discriminative features and decodes MI tasks effectively, improving the performance of MI-BCIs. Our findings could provide significant potential for improving the clinical application and development of MI-based BCI systems.

Motor Imagery (MI)-based Brain Computer Interface (BCI) system is a potential technology for active neurorehabilitation of stroke patients by complementing the conventional passive rehabilitation methods. Research to date mainly focused on classifying left vs. right hand/foot MI of stroke patients. Though a very few studies have reported decoding imagined hand movement directions using electroencephalogram (EEG)-based BCI, the experiments were conducted on healthy subjects. Our work analyzes MI-based brain cortical activity from EEG signals and decodes the imagined hand movement directions in stroke patients. The decoded direction (left vs. right) of hand movement imagination is used to provide control commands to a motorized arm support on which patient's affected (paralyzed) arm is placed. This enables the patient to move his/her stroke-affected hand towards the intended (imagined) direction that aids neuroplasticity in the brain. The synchronization measure called Phase Locking Value (PLV), extracted from EEG, is the neuronal signature used to decode the directional movement of the MI task. Event-related desynchronization/synchronization (ERD/ERS) analysis on Mu and Beta frequency bands of EEG is done to select the time bin corresponding to the MI task. The dissimilarities between the two directions of MI tasks are identified by selecting the most significant channel pairs that provided maximum difference in PLV features. The training protocol has an initial calibration session followed by a feedback session with 50 trials of MI task in each session. The feedback session extracts PLV features corresponding to most significant channel pairs which are identified in the calibration session and is used to predict the direction of MI task in left/right direction. An average MI direction classification accuracy of 74.44% is obtained in performing the training protocol and 68.63% from the prediction protocol during feedback session on 16 stroke patients.

Existing research methods for recognizing EEG (Electroencephalogram) signals in motor imagery (MI) often overlook the dynamic changes of brain networks over time, resulting in insufficient classification accuracy for MI tasks. This article addresses the recognition problem of dynamic changes in brain networks during MI tasks and applies an EEG signal classification method based on multi-layer time-varying functional brain networks. This article uses the BCI (Brain-Computer Interface) Competition IV 2a dataset to preprocess the raw EEG signals through bandpass filtering and CSP (Common Spatial Pattern) algorithm. The EEG signals of the MI task are divided into 7 1-second time windows with a step size of 0.5 seconds. Within each time window, Pearson correlation coefficients between EEG channels can be calculated to generate corresponding brain networks, and multiple time-varying functional brain networks can be constructed by stacking the brain networks from multiple time windows. The network topology features, node degree, clustering coefficient, network efficiency, and multi-layer network features of each window can be extracted, including Multiplex Clustering Coefficient (MCC), Multiplex Participation Coefficient (MPC), and inter layer correlation coefficient. By dividing the dataset through 10 fold cross validation, the random forest algorithm can be used to classify and recognize four types of motion imagination tasks. The experimental results show that the average recognition rate of the article’s method in four types of MI tasks reached 89.19%. This method can improve the classification accuracy of MI tasks and enhance a comprehensive understanding of the dynamic changes in brain networks during the process of MI.

The aim of this paper is to classify both (left and right) hand motor imagery (MI) tasks using event-related (de) synchronization (ERD/ERS) patterns based on the inter trial variance (IV). In this work, publicly available online BCI-2003 competition data set was used which contains MI based Electroencephalography (EEG) data of a single subject. The raw MI based EEG signals were extracted from C3 and C4 channels and band pass filtered to mu (8–12 Hz) frequency band. The mean, inter trial variance and ERD/ERS patterns were calculated in both the channels for both MI tasks of the subject. The patterns were used as feature vector and input to the different machine learning classifiers. The logistic regression and Naive Bayesian classifiers were used for the classification of both MI tasks of the subject. For logistic regression classifier, the accuracy, precision and sensitivity were found to be 92.40%, 92.00% and 92.00%, respectively. Meanwhile, 92.19% accuracy, 92.00% precision and 92.00% sensitivity were obtained for the Naive Bayesian classifier. The logistic regression classifier provides maximum classification accuracy (92.40%) in comparison with the existing methods (87.90–90.20%). Result shows that, the proposed method can quantify ERD/ERS patterns effectively in both the channels and also classifies both MI tasks of the subject successfully.

Dealing with subjects who are unable to attain a proper level of performance, that is, those with brain–computer interface (BCI) illiteracy or BCI inefficients, is still a major issue in human electroencephalography (EEG) BCI systems. The most suitable approach to address this issue is to analyze the EEG signals of individual subjects independently recorded before the main BCI tasks, to evaluate their performance on these tasks. This study mainly focused on non-linear analyses and deep learning techniques to investigate the significant relationship between the intrinsic characteristics of a prior idle resting state and the subsequent BCI performance. To achieve this main objective, a public EEG motor/movement imagery dataset that constituted two individual EEG signals recorded from an idle resting state and a motor imagery BCI task was used in this study. For the EEG processing in the prior resting state, spectral analysis but also non-linear analyses, such as sample entropy, permutation entropy, and recurrent quantification analyses (RQA), were performed to obtain individual groups of EEG features to represent intrinsic EEG characteristics in the subject. For the EEG signals in the BCI tasks, four individual decoding methods, as a filter-bank common spatial pattern-based classifier and three types of convolution neural network-based classifiers, quantified the subsequent BCI performance in the subject. Statistical linear regression and ANOVA with post hoc analyses verified the significant relationship between non-linear EEG features in the prior resting state and three types of BCI performance as low-, intermediate-, and high-performance groups that were statistically discriminated by the subsequent BCI performance. As a result, we found that the frontal theta rhythm ranging from 4 to 8 Hz during the eyes open condition was highly associated with the subsequent BCI performance. The RQA findings that higher determinism and lower mean recurrent time were mainly observed in higher-performance groups indicate that more regular and stable properties in the EEG signals over the frontal regions during the prior resting state would provide a critical clue to assess an individual BCI ability in the following motor imagery task.

Multi-channel Electroencephalograph (EEG) signal is an important source of neural information for motor imagery (MI) limb movement intent decoding. The decoded MI movement intent often serve as potential control input for brain-computer interface (BCI) based rehabilitation robots. However, the presence of multiple dynamic artifacts in EEG signal leads to serious processing challenge that affects the BCI system in practical settings. Hence, this study propose a hybrid approach based on Low-rank spatiotemporal filtering technique for concurrent elimination of multiple EEG artifacts. Afterwards, a convolutional neural network based deep learning model (ConvNet-DL) that extracts neural information from the cleaned EEG signal for MI tasks decoding was built. The proposed method was studied in comparison with existing artifact removal methods using EEG signals of transhumeral amputees who performed five different MI tasks. Remarkably, the proposed method led to significant improvements in MI task decoding accuracy for the ConvNet-DL model in the range of 8.00~13.98%, while up to 14.38% increment was recorded in terms of the MCC: Mathew correlation coefficients at p<0.05. Also, a signal to error ratio of more than 11 dB was recorded by the proposed method.Clinical Relevance- This study showed that a combination of the proposed hybrid EEG artifact removal method and ConvNet-DL can significantly improve the decoding accuracy of MI upper limb movement tasks. Our findings may provide potential control input for BCI rehabilitation robotic systems.

Riemannian geometry-based classification (RGBC) gained popularity in the field of brain-computer interfaces (BCIs) lately, due to its ability to deal with non-stationarities arising in electroencephalography (EEG) data. Domain adaptation, however, is most often performed on sample covariance matrices (SCMs) obtained from EEG data, and thus might not fully account for components affecting covariance estimation itself, such as regional trends. Detrended cross-correlation analysis (DCCA) can be utilized to estimate the covariance structure of such signals, yet it is computationally expensive in its original form. A recently proposed online implementation of DCCA, however, allows for its fast computation and thus makes it possible to employ DCCA in real-time applications. In this study we propose to replace the SCM with the DCCA matrix as input to RGBC and assess its effect on offline and online BCI performance. First we evaluated the proposed decoding pipeline offline on previously recorded EEG data from 18 individuals performing left and right hand motor imagery (MI), and benchmarked it against vanilla RGBC and popular MI-detection approaches. Subsequently, we recruited eight participants (with previous BCI experience) who operated an MI-based BCI (MI-BCI) online using the DCCA-enhanced Riemannian decoder. Finally, we tested the proposed method on a public, multi-class MI-BCI dataset. During offline evaluations the DCCA-based decoder consistently and significantly outperformed the other approaches. Online evaluation confirmed that the DCCA matrix could be computed in real-time even for 22-channel EEG, as well as subjects could control the MI-BCI with high command delivery (normalized Cohen's κ: 0.7409 ± 0.1515) and sample-wise MI detection (normalized Cohen's κ: 0.5200 ± 0.1610). Post-hoc analysis indicated characteristic connectivity patterns under both MI conditions, with stronger connectivity in the hemisphere contralateral to the MI task. Additionally, fractal scaling exponent of neural activity was found increased in the contralateral compared to the ipsilateral motor cortices (C4 and C3 for left and right MI, respectively) in both classes. Combining DCCA with Riemannian geometry-based decoding yields a robust and effective decoder, that not only improves upon the SCM-based approach but can also provide relevant information on the neurophysiological processes behind MI.

The brain-computer interface (BCI) work is to provide humans an alternative channel that allows direct transmission of messages from the brain by analyzing the brain’s mental activities [1–7]. The brain activity is recorded by means of multi-electrode electroencephalographic (EEG) signals that are either invasive or noninvasive. Noninvasive recording is convenient and popular in BCI applications so it is commonly used. According to the definition suggested at the first international meeting for BCI technology, the term BCI is reserved for a system that must not depend on the brain’s normal output pathways of peripheral nerves and muscles [2]. It has become popular for BCI systems on motor imagery (MI) EEG signals in the last decade [8]. It reveals that there are special characteristics of event-related desynchronization (ERD) and synchronization (ERS) in mu and beta rhythms over the sensorimotor cortex during MI tasks by discriminating EEG signals between left and right MIs [9, 10]. ERD/ERS is the task-related or event-related change in the amplitude of the oscillatory behavior of specific cortical areas within various frequency bands. An amplitude (or power) increase is defined as event-related synchronization while an amplitude (or power) decrease is defined as event-related desynchronization. As other event-related potentials, ERD/ERS patterns are associated with sensory processing and motor behavior [2]. The principal objective of this study is to propose a BCI system, which combines neuro-fuzzy prediction and multiresolution fractal feature vectors (MFFVs) with support vector machine, for MI classification.

Brain computer interfaces (BCIs) based on motor imagery (MI) play an important role in helping to improve and restore the loss of physical function. However, traditional MI-BCIs are limited to the motion intention of gross limb, which places many restrictions on their applications. This study proposes a hybrid paradigm based on MI and the steady-state somatosensory evoked potential, with the aim of improving the spatial resolution of MI recognition. Twelve subjects participated in this study. They performed MI tasks under MI and hybrid conditions. In the MI condition, subjects only performed MI tasks, whereas, in the hybrid condition, they received an electrical stimulus while performing the same tasks. Under the hybrid condition, subjects were required not to pay extra attention to the electrical stimulation. The MI task included imagining clenching the right hand and lifting the right forearm. The classifier was built using the filter bank common spatial pattern algorithm and a support vector machine, and online experiments were used to verify the recognition of two MI tasks. During the online experiments, all subjects were able to output different commands at a recognition accuracy far higher than the random level. The average classification accuracy of the hybrid condition reached 76.39%, with a maximum value of 88.34%, which was about 11% higher than that of the MI condition. Moreover, based on offline data, the classification performance using the event-related desynchronization (ERD) feature under the hybrid condition did not differ significantly from that under the MI condition, indicating that the introduction of electrical stimulation did not interfere in the separability of ERD. The proposed paradigm improved the efficiency of decoding multiple MI locations within a single limb. Despite the introduction of external stimuli, users could still drive the new system in the same way as MI in traditional MI-BCI.

Brain-Computer Interface (BCI) systems allow user to operate devices without muscular activation and experimental results indicate that they can induce activity-dependent plasticity. A BCI system has two key features: exploitation of brain signal changes induced by the cognitive task assigned to the user and mutual learning. With respect to the first feature, BCI users can learn to control and hence change their cortical activity, for example by performing a motor imagery task. Mutual learning, on the other hand, is set up between users, which have to learn how to control their brain activity, and the classifier used for learning and categorize brain signal changes. This mutual learning can be mediated by the use of feedback, which in turn can be discrete or continuous, abstract or realistic and so on. Motor imagery represents a promising approach for stroke rehabilitation, thus a BCI based on a motor imagery paradigm could be a useful tool. Nevertheless, in stroke rehabilitation settings two issues are emerging which are also crucial for motor imagery BCIs. The first one relates to the importance of recognize the imagery ability of the subject, the second one to the importance of feedback in re-learning of motor skills. The present study relates to this second issue and it is aimed at investigating attitudes of BCI users towards feedback. We considered a sample of six healthy males, using an EEG-based BCI and executing a Motor Imagery task without feedback (training phase) and with feedback (performance phase). A horizontal bar of varying length depicted on a computer screen was used as feedback medium and overall accuracy data were matched with neurophysiological data acquired from P z and O z . Results could be of importance for BCI use as neurorehabilitation tool in stroke rehabilitation protocols since they suggest that some subjects could need an increased cortical activation in order to deal with a stimulus like visual feedback. Moreover, they reveal a possible association between worsening in performance and increased cortical activation needed in order to cope with feedback.

In recent years, deep-learning-based motor imagery (MI) electroencephalography (EEG) decoding methods have shown great potential in the field of the brain–computer interface (BCI). The existing literature is relatively mature in decoding methods for two classes of MI tasks. However, with the increase in MI task classes, decoding studies for four classes of MI tasks need to be further explored. In addition, it is difficult to obtain large-scale EEG datasets. When the training data are limited, deep-learning-based decoding models are prone to problems such as overfitting and poor robustness. In this study, we design a data augmentation method for MI-EEG. The original EEG is slid along the time axis and reconstructed to expand the size of the dataset. Second, we combine the gated recurrent unit (GRU) and convolutional neural network (CNN) to construct a parallel-structured feature fusion network to decode four classes of MI tasks. The parallel structure can avoid temporal, frequency and spatial features interfering with each other. Experimenting on the well-known four-class MI dataset BCI Competition IV 2a shows a global average classification accuracy of 80.7% and a kappa value of 0.74. The proposed method improves the robustness of deep learning to decode small-scale EEG datasets and alleviates the overfitting phenomenon caused by insufficient data. The method can be applied to BCI systems with a small amount of daily recorded data.

A brain-computer interface (BCI) based on motor imagery (MI) from the same limb can provide an intuitive control pathway but has received limited attention. It is still a challenge to classify multiple MI tasks from the same limb. The goal of this study is to propose a novel decoding method to classify the MI tasks of four joints of the same upper limb and the resting state. EEG signals were collected from 20 participants. A time-distributed attention network (TD-Atten) was proposed to adaptively assign different weights to different classes and frequency bands of the input multiband Common Spatial Pattern (CSP) features. The long short-term memory (LSTM) and dense layers were then used to learn sequential information from the reweight features and perform the classification. Our proposed method outperformed other baseline and deep learning-based methods and obtained the accuracies of 46.8% in the 5-class scenario and 53.4% in the 4-class scenario. The visualization results of attention weights indicated that the proposed framework can adaptively pay attention to alpha-band related features in MI tasks, which was consistent with the analysis of brain activation patterns. These results demonstrated the feasibility and interpretability of the attention mechanism in MI decoding and the potential of this fine MI paradigm to be applied for the control of a robotic arm or a neural prosthesis.

The right and left-hand motor imagery (MI) analysis based on the electroencephalogram (EEG) signal can directly link the central nervous system to a computer or a device. This study aims to identify a set of robust and nonlinear effective brain connectivity features quantified by transfer entropy (TE) to characterize the relationship between brain regions from EEG signals and create a hierarchical feature selection and classification for discrimination of right and lefthand MI tasks. TE is calculated among EEG channels as the distinctive, effective connectivity features. TE is a model-free method that can measure nonlinear effective connectivity and analyze multivariate dependent directed information flow among neural EEG channels. Then four feature subset selection methods namely relief-F, Fisher, Laplacian, and local learningbased clustering (LLCFS) algorithms are used to choose the most significant effective connectivity features and reduce redundant information. Finally, support vector machine (SVM) and linear discriminant analysis (LDA) methods are used for classification. Results show that the best performance in 29 healthy subjects and 60 trials is achieved using the TE method via the Relief-F algorithm as feature selection and support vector machine (SVM) classification with 91.02% accuracy. The TE index and a hierarchical feature selection and classification can be useful for the discrimination of right- and left-hand MI tasks from multichannel EEG signals. Effective connectivity features were extracted from electroencephalogram (EEG) to analyze relationships between regions.Four feature selection methods used to select most significant effective features.Support vector machine (SVM) used for discrimination of right and left hand motor imagery (MI) task. In this study, we investigated brain activity using effective connectivity during MI task based on EEG signals. The motor imagery task can accomplish the same goal as motor execution, since they are both activated by the same brain area. Transfer entropy, coherence, and Granger casualty were employed to extract the features. Differential patterns of activity between the left vs. right MI task showed activity around the motor area rather than other areas. In order to reduce redundant information and select the most significant effective connectivity features, four feature subset selection algorithms are used: Relief-F, Fisher, Laplacian, and learning-based clustering feature selection (LLCFS). Then, support vector machine (SVM) and linear discriminant analysis (LDA) are used to classify left and right hand MI task. Comparison of three different connectivity methods showed that TE index had the highest classification accuracy, and could be useful for the discrimination of right and left hand MI tasks from multichannel EEG signals.