EEG data analysis with stacked differentiable neural computers

This chapter examines the use of Generative Adversarial Networks (GANs) in analyzing electroencephalogram (EEG) data. EEG is an electrophysiological method that records brain activity. EEG is used to diagnose neurological disorders and is also very important for brain-computer interface (BCI) systems. Although EEG data processing and analysis is widely used, it faces some difficulties, which reveals the necessity of advanced signal processing techniques. GANs, on the other hand, are advanced machine learning techniques and play an essential role in EEG data analysis. GANs are known for their ability to produce synthetic data similar to actual data, and this feature provides significant advantages in the analysis of EEG data. In particular, GANs are effective at filtering noise, improving data quality, and generating synthetic data. Given the complexity and diversity of EEG data, caution must be exercised in training GAN models and the accuracy of synthetic data. Current limitations of GANs in EEG data analysis and ongoing research to overcome these limitations are also examined.

Although the potential of the powerful mapping and representational capabilities of recurrent network architectures is generally recognized by the neural network research community, recurrent neural networks have not been widely used for the control of nonlinear dynamical systems, possibly due to the relative ineffectiveness of simple gradient descent training algorithms. Developments in the use of parameter-based extended Kalman filter algorithms for training recurrent networks may provide a mechanism by which these architectures will prove to be of practical value. This paper presents a decoupled extended Kalman filter (DEKF) algorithm for training of recurrent networks with special emphasis on application to control problems. We demonstrate in simulation the application of the DEKF algorithm to a series of example control problems ranging from the well-known cart-pole and bioreactor benchmark problems to an automotive subsystem, engine idle speed control. These simulations suggest that recurrent controller networks trained by Kalman filter methods can combine the traditional features of state-space controllers and observers in a homogeneous architecture for nonlinear dynamical systems, while simultaneously exhibiting less sensitivity than do purely feedforward controller networks to changes in plant parameters and measurement noise.

Deep neural networks (DNNs) have shown a great achievement in acoustic modeling for speech recognition task. Of these networks, convolutional neural network (CNN) is an effective network for representing the local properties of the speech formants. However, CNN is not suitable for modeling the long-term context dependencies between speech signal frames. Recently, the recurrent neural networks (RNNs) have shown great abilities for modeling long-term context dependencies. However, the performance of RNNs is not good for low-resource speech recognition tasks, and is even worse than the conventional feed-forward neural networks. Moreover, these networks often overfit severely on the training corpus in the low-resource speech recognition tasks. This paper presents the results of our contributions to combine CNN and conventional RNN with gate, highway, and residual networks to reduce the above problems. The optimal neural network structures and training strategies for the proposed neural network models are explored. Experiments were conducted on the Amharic and Chaha datasets, as well as on the limited language packages (10-h) of the benchmark datasets released under the Intelligence Advanced Research Projects Activity (IARPA) Babel Program. The proposed neural network models achieve 0.1–42.79% relative performance improvements over their corresponding feed-forward DNN, CNN, bidirectional RNN (BRNN), or bidirectional gated recurrent unit (BGRU) baselines across six language collections. These approaches are promising candidates for developing better performance acoustic models for low-resource speech recognition tasks.

Due to the simple design pipeline, end-to-end (E2E) neural models for speech enhancement (SE) have attracted great interest. In order to improve the performance of the E2E model, the local and sequential properties of speech should be efficiently taken into account when modelling. However, in most current E2E models for SE, these properties are either not fully considered or are too complex to be realized. In this letter, we propose an efficient E2E SE model, termed WaveCRN. Compared with models based on convolutional neural networks (CNN) or long short-term memory (LSTM), WaveCRN uses a CNN module to capture the speech locality features and a stacked simple recurrent units (SRU) module to model the sequential property of the locality features. Different from conventional recurrent neural networks and LSTM, SRU can be efficiently parallelized in calculation, with even fewer model parameters. In order to more effectively suppress noise components in the noisy speech, we derive a novel restricted feature masking approach, which performs enhancement on the feature maps in the hidden layers; this is different from the approaches that apply the estimated ratio mask to the noisy spectral features, which is commonly used in speech separation methods. Experimental results on speech denoising and compressed speech restoration tasks confirm that with the SRU and the restricted feature map, WaveCRN performs comparably to other state-of-the-art approaches with notably reduced model complexity and inference time.

Finite-time synchronization control of a class of memristor-based recurrent neural networks

In this paper, for the first time, the deep gated recurrent unit (Deep GRU) is used as a new macromodeling approach for nonlinear circuits. Similar to Long Short-Term Memory (LSTM), the GRU has gating units that control the information flow and makes the network less prone to the vanishing gradient problem. Having a smaller number of gates causes GRU to have fewer parameters compared to LSTM leading to better model accuracy. Using the gates leads gradient formulations to have additive nature which helps them to be more resistant to vanishing and consequently learn long sequences of data. The proposed macromodeling method is capable of modeling nonlinear circuits more accurately and using fewer parameters compared to the conventional LSTM macromodeling method. To further improve the GRU performance, a regularization technique called Gaussian dropout is applied in this paper on deep GRU (GDGRU) to reduce the overfitting problem resulting in better test error. Additionally, the models obtained from the proposed techniques are remarkably faster than the original transistor-level models. To verify the superiority of the proposed method, time-domain modeling of three nonlinear circuits is provided. For these circuits, the comparisons of the accuracy and speed between the conventional recurrent neural network (RNN), the LSTM, and the proposed macromodeling methods are provided.

Mining Electronic Health Records (EHRs) is of great significance to improve the efficiency and quality of medical services. In recent years, researchers have used deep recurrent neural networks to predict patients’ next-period prescriptions. The main challenges of predicting next-period prescriptions are as follows: i) The latent interdependence information which can be utilized to enhance the representation of the data between heterogeneous features is dynamic. However, most existing approaches do not consider capturing and utilizing this information. ii) Conventional recurrent neural networks cannot store historical interdependent information between heterogeneous features and take advantage of it. To address these challenges, We propose a novel attention mechanism named Cross Attention (CA) that can capture the interdependence between two sequences. We further propose three types of recurrent neural networks that can capture and utilize current and historical latent interdependence between the two sequences. Extensive experiments on real-world data demonstrate that DSCA networks can capture the interdependence information between sequences and outperform state-of-the-art methods.

Prediction and optimisation of fuel consumption for inland ships considering real-time status and environmental factors

In blast furnace (BF) ironmaking process, the silicon content of hot metal is regarded as an important index to evaluate the stability of blast furnace and the quality of pig iron. In reality, the reactions inside BF with the features of dynamic, nonlinear and hysteretic impede accurate mathematical modeling. Data-driven models predict silicon content variation tendency through the analysis of historical data, which have achieved beneficial results. However, existing data-driven models are inefficient to handle time series specifically. Therefore, this paper proposes a recurrent neural network (RNN) which is apt at processing sequential data for silicon content prediction. A Long short-term memory (LSTM) unit is introduced to solve long-term dependency problems in conventional recurrent neural networks. Experiment results indicate that LSTM-RNN shows stronger adaptability and higher prediction accuracy at 86%-94% than that of other data-driven models, which is of great significance for controling the silicon content of hot metal.

Matrix-based wavelet transformation embedded in recurrent neural networks for wind speed prediction

Individual choices and preferences are important factors that impact decision making. Artificial intelligence can predict decisions by objectively detecting individual choices and preferences using natural language processing, computer vision, and machine learning. Brain–computer interfaces can measure emotional reactions and identify brain activity changes linked to positive or negative emotions, enabling more accurate prediction models. This research aims to build an individual choice prediction system using electroencephalography (EEG) signals from the Shanghai Jiao Tong University emotion and EEG dataset (SEED). Using EEG, we built different deep learning models, such as a convolutional neural network, long short-term memory (LSTM), and a hybrid model to predict choices driven by emotional stimuli. We also compared their performance with different classical classifiers, such as k-nearest neighbors, support vector machines, and logistic regression. We also utilized ensemble classifiers such as random forest, adaptive boosting, and extreme gradient boosting. We evaluated our proposed models and compared them with previous studies on SEED. Our proposed LSTM model achieved good results, with an accuracy of 96%.

Tropical cyclones are among the most powerful and destructive meteorological systems on Earth. In this paper, we propose a novel deep learning model for tropical cyclone track prediction method. Specifically, the track task is regarded as a time series predicting challenge, and then a deep learning framework by a bidirectional gate recurrent unit network (BiGRU) with attention mechanism is developed for track prediction. This proposed model can excavate the effective information of the historical track in a deeper and more accurate way. Data experiments are conducted on tropical cyclone best-track data provided by the Joint Typhoon Warning Center (JTWC) from 1988 to 2017 in the northwestern Pacific Ocean. Results show that our model performs well for tracks of 6, 12, 24, 48, and 72 h in the future. The prediction results show that our proposed combined model is superior to state-of-the-art deep learning models, including a recurrent neural network (RNN), long short-term memory neural network (LSTM), gate recurrent unit network (GRU), and BiGRU without the use of attention mechanism. In comparison with the methods used by the China Meteorological Administration, Japan Meteorological Agency, and the JTWC, our method has obvious advantages in the mid- to long-term track forecasting, especially in the next 72 h.

This paper proposes a feed-forward neural network predictor to adaptively estimate and suppress the narrowband interference (NBI) in the Direct Sequence-Code Division Multiple Access (DS-CDMA) signal. The iterative code-aided estimation is used to further improve the system performance. Simulation results reveal that the proposed algorithm outperforms conventional linear prediction filtering and recurrent neural networks (RNN) based NBI rejection methods, in different interference models.

To apply deep learning algorithms using a conventional convolutional neural network (CNN) and a recurrent CNN to differentiate three breast cancer molecular subtypes on MRI. A total of 244 patients were analyzed, 99 in training dataset scanned at 1.5 T and 83 in testing-1 and 62 in testing-2 scanned at 3 T. Patients were classified into 3 subtypes based on hormonal receptor (HR) and HER2 receptor: (HR+/HER2-), HER2+, and triple negative (TN). Only images acquired in the DCE sequence were used in the analysis. The smallest bounding box covering tumor ROI was used as the input for deep learning to develop the model in the training dataset, by using a conventional CNN and the convolutional long short-term memory (CLSTM). Then, transfer learning was applied to re-tune the model using testing-1(2) and evaluated in testing-2(1). In the training dataset, the mean accuracy evaluated using tenfold cross-validation was higher by using CLSTM (0.91) than by using CNN (0.79). When the developed model was applied to the independent testing datasets, the accuracy was 0.4-0.5. With transfer learning by re-tuning parameters in testing-1, the mean accuracy reached 0.91 by CNN and 0.83 by CLSTM, and improved accuracy in testing-2 from 0.47 to 0.78 by CNN and from 0.39 to 0.74 by CLSTM. Overall, transfer learning could improve the classification accuracy by greater than 30%. The recurrent network using CLSTM could track changes in signal intensity during DCE acquisition, and achieved a higher accuracy compared with conventional CNN during training. For datasets acquired using different settings, transfer learning can be applied to re-tune the model and improve accuracy. • Deep learning can be applied to differentiate breast cancer molecular subtypes. • The recurrent neural network using CLSTM could track the change of signal intensity in DCE images, and achieved a higher accuracy compared with conventional CNN during training. • For datasets acquired using different scanners with different imaging protocols, transfer learning provided an efficient method to re-tune the classification model and improve accuracy.

The removal of Ocular Artifacts (OA) in Electroencephalogram (EEG) data is one of the key challenges in the analysis of brain recordings. Brain activity produces electroencephalogram signals, which consists of some of vital signs of neurological disorders such as epilepsy, tumor cerebrovascular lesions and the problems associated with the trauma. These signals can be acquired by placing the electrodes on the scalp at specified positions and exists in order of 1–5μv, whose frequency range is DC-64 Hz. Acquisition of these signals mainly suffers from different unwanted signals (artifacts or noise) resulting in less signal information for identification. In this paper, two algorithms are proposed namely, Multi Scale Principal Component Analysis (MSPCA) and Iterative Soft Thresholding (IST) using wavelets in removing the Ocular Artifacts (OA) present in the EEG signals. This paper discusses not only the performance comparison of two algorithms on statistical parameters of EEG signals such as Signal to Noise Ratio, (SNR), SNRI or Noise Figure (NF) and Absolute Average Error (AAE) but also estimated the run time of each algorithm i.e., computational time of each algorithm.