Dynamic graph neural network with adaptive edge attributes for air quality prediction: A case study in China

Due to the objectivity of emotional expression in the central nervous system, EEG-based emotion recognition can effectively reflect humans' internal emotional states. In recent years, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have made significant strides in extracting local features and temporal dependencies from EEG signals. However, CNNs ignore spatial distribution information from EEG electrodes; moreover, RNNs may encounter issues such as exploding/vanishing gradients and high time consumption. To address these limitations, we propose an attention-based temporal graph representation network (ATGRNet) for EEG-based emotion recognition. Firstly, a hierarchical attention mechanism is introduced to integrate feature representations from both frequency bands and channels ordered by priority in EEG signals. Second, a graph convolutional neural network with top-k operation is utilized to capture internal relationships between EEG electrodes under different emotion patterns. Next, a residual-based graph readout mechanism is applied to accumulate the EEG feature node-level representations into graph-level representations. Finally, the obtained graph-level representations are fed into a temporal convolutional network (TCN) to extract the temporal dependencies between EEG frames. We evaluated our proposed ATGRNet on the SEED, DEAP and FACED datasets. The experimental findings show that the proposed ATGRNet surpasses the state-of-the-art graph-based mehtods for EEG-based emotion recognition.

With the development of precision sensing instruments and data storage devices, the fusion of multi-sensor data in gearbox fault diagnosis has attracted much attention. However, existing methods have difficulty in capturing the local temporal dependencies of multi-sensor monitoring information, and the inescapable noise severely decreases the accuracy of multi-sensor information fusion diagnosis. To address these issues, this paper proposes a fault diagnosis method based on dynamic graph convolutional neural networks and hard threshold denoising. Firstly, considering that the relationships between monitoring data from different sensors change over time, a dynamic graph structure is adopted to model the temporal dependencies of multi-sensor data, and, further, a graph convolutional neural network is constructed to achieve the interaction and feature extraction of temporal information from multi-sensor data. Secondly, to avoid the influence of noise in practical engineering, a hard threshold denoising strategy is designed, and a learnable hard threshold denoising layer is embedded into the graph neural network. Experimental fault datasets from two typical gearbox fault test benches under environmental noise are used to verify the effectiveness of the proposed method in gearbox fault diagnosis. The experimental results show that the proposed DDGCN method achieves an average diagnostic accuracy of up to 99.7% under different levels of environmental noise, demonstrating good noise resistance.

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting

Messenger RNA (mRNA) has emerged as a critical global technology that requires global joint efforts from different entities to develop a COVID-19 vaccine. However, the chemical properties of RNA pose a challenge in utilizing mRNA as a vaccine candidate. For instance, the molecules are prone to degradation, which has a negative impact on the distribution of mRNA among patients. In addition, little is known of the degradation properties of individual RNA bases in a molecule. Therefore, this study aims to investigate whether a hybrid deep learning can predict RNA degradation from RNA sequences. Two deep hybrid neural network models were proposed, namely GCN_GRU and GCN_CNN. The first model is based on graph convolutional neural networks (GCNs) and gated recurrent unit (GRU). The second model is based on GCN and convolutional neural networks (CNNs). Both models were computed over the structural graph of the mRNA molecule. The experimental results showed that GCN_GRU hybrid model outperform GCN_CNN model by a large margin during the test time. Validation of proposed hybrid models is performed by well-known evaluation measures. Among different deep neural networks, GCN_GRU based model achieved best scores on both public and private MCRMSE test scores with 0.22614 and 0.34152, respectively. Finally, GCN_GRU pre-trained model has achieved the highest AuC score of 0.938. Such proven outperformance of GCNs indicates that modeling RNA molecules using graphs is critical in understanding molecule degradation mechanisms, which helps in minimizing the aforementioned issues. To show the importance of the proposed GCN_GRU hybrid model, in silico experiments has been contacted. The in-silico results showed that our model pays local attention when predicting a given position’s reactivity and exhibits interesting behavior on neighboring bases in the sequence.

Event-related potentials (ERPs) are a superposition of electric potential differences generated by neurophysiological activity associated with psychophysical events. Spatiotemporal dissociation of these signal sources can supplement conventional ERP analysis and improve source localization. However, results from established source separation methods applied to ERPs can be challenging to interpret. Hence, we have developed a recurrent neural network (RNN) method for blind source separation. The RNN transforms input step pulse signals representing events into corresponding ERP difference waveforms. Source waveforms are obtained from penultimate layer units and scalp maps are obtained from feed-forward output layer weights that project these source waveforms onto EEG electrode amplitudes. An interpretable, sparse source representation is achieved by incorporating L1 regularization of signals obtained from the penultimate layer of the network during training. This RNN method was applied to four ERP difference waveforms (MMN, N170, N400, P3) from the open-access ERP CORE database, and independent component analysis (ICA) was applied to the same data for comparison. The RNN decomposed these ERPs into eleven spatially and temporally separate sources that were less noisy, tended to be more ERP-specific, and were less similar to each other than ICA-derived sources. The RNN sources also had less ambiguity between source waveform amplitude, scalp potential polarity, and equivalent current dipole orientation than ICA sources. In conclusion, the proposed RNN blind source separation method can be effectively applied to grand-average ERP difference waves and holds promise for further development as a computational model of event-related neural signals.

The Sixth International Symposium on Neural Networks (ISNN 2009)

Reservoir simulation results currently provide the basis for important reservoir engineering decisions; grid complexity and non-linearity of these models demand high computational time and memory. The physics-based simulation process must be repeated to increase model prediction accuracy or to perform history matching; consequently, the simulation process is often time-consuming. This paper describes a new methodology based on a deep neural network (DNN) technique, the graph convolutional neural network (G-CNN). G-CNN increases the modeling prediction speed and efficiency by compressing the computational time and memory usage of the reservoir simulation. A G-CNN model was used to perform the reservoir simulations described. This new methodology combines physics-based and data-driven models in reservoir simulation. The workflow generates training datasets, enabling intelligent sampling of the reservoir production data in the G-CNN training process. Bottomhole pressure constraints were set for all simulations. The production data generated by the reservoir model, with the mesh connectivity information, is used to generate the G-CNN model. This approach can be viewed as hybrid data-driven, retaining the underlying physics of the reservoir simulator. The resulting G-CNN model can perform reservoir simulations for any computational grid and production time horizon. The method uses convolutional neural network and mesh connections in a fully differentiable scheme to compress the simulation state size and learns the reservoir dynamics on this compressed form. G-CNN analysis was performed on an Eagle Ford-type reservoir model. The transmissibility, pore volume, pressure, and saturation at an initial time state were used as input features to predict final pressure and saturations. Prediction accuracy of 95% was obtained by hyperparameter tuning off the G-CNN architecture. By compressing the simulation state size and learning the time-dependent reservoir dynamics on this compressed representation, reservoir simulations can be emulated by a graph neural network which uses significantly less computation and memory. The G-CNN model can be used on any computational grid because it preserves the structure of the physics. The G-CNN model is trained by mapping an initial time state to a future prediction time state; consequently, the model can be used for generalizing predictions in any grid sizes and time steps while maintaining accuracy. The result is a computationally and memory efficient neural network that can be iterated and queried to reproduce a reservoir simulation. This novel methodology combines field-scale physics-based reservoir modeling and DNN for reservoir simulations. The new reservoir simulation workflow, based on the G-CNN model, maps the time state predictions in a resampled grid, reducing computational time and memory. The new methodology presents a general method for compressing reservoir simulations, assisting in fast and accurate production forecasting.

This is a continuation of a recent study (Doruk RO, Zhang K. Fitting of dynamic recurrent neural network models to sensory stimulus-response data. J Biol Phys 2018; 44: 449-469), where a continuous time dynamical recurrent neural network is fitted to neural spiking data. In this research, we address the issues arising from the inclusion of sigmoidal gain function parameters to the estimation algorithm. The neural spiking data will be obtained from the same model as that of Doruk and Zhang, but we propose a different model for identification. This will also be a continuous time recurrent neural network, but with generic sigmoidal gains. The simulation framework and estimation algorithms are kept similar to that of Doruk and Zhang so that we can have a solid base to compare the results. We evaluate the estimation performance in two different ways. First, we compare the firing rate responses of the original and the estimated model. We find that responses of both models to the same stimuli are similar. Secondly, we evaluate variations of the standard deviations of the estimates against a number of samples and stimulus parameters. They show a similar pattern to that of Doruk and Zhang. We thus conclude that our model serves as a reasonable alternative provided that firing rate is the response of interest (to any stimulus).

We introduced a novel methodology Multivariate Degree Distribution to Dynamic Graph (MDD2DG) with Inter-channel Relevance Attention (IRA) mechanism to analyze multi-channel Electrocardiogram (ECG) signals and explore signal connections across different channels. Our methodology comprises three main steps. First, multi-channel cardiac signals are transformed into multi-channel visual graphs to extract crucial degree distribution features. Then, degree distributions are mapped into dynamic graphs using a neural network with an IRA mechanism. After that, critical features are extracted within dynamic graphs utilizing a Graph Convolutional Neural Networks (GCNNs), and classification is subsequently performed using a multilayer perceptron. In this model, a method of multi-scale position embedding was introduced, which significantly enhanced the processing efficiency of the model by providing a simpler yet sufficiently effective feature representation. Compared to traditional complex network methods, our approach replaces fixed formula-calculated features with dynamic graph models, resulting in improved recognition accuracy. In the experiments, we achieved an impressive 99.94% classification accuracy for distinguishing ECG signals from the five distinct locations (AMI, ASMI, ALMI, IMI and ILMI) with myocardial infarction (MI) as well as those of the healthy controls (HC). This work contributes to the analysis of complex physiological signals in the field of multi-channel ECG sequence, and provides a robust approach with promising implications for improving clinical medicine and the early detection of cardiac diseases.

Nowadays , nonlinear time series and artificial neural networks (ANN) models are used for forecasting in the field of business, agriculture and soon. Recent studies have shown, ANN have been successfully used for forecasting of financial and agriculture data series The classical methods used for time series prediction like Box-Jenkins or ARIMA assumes that there is a linear relationship between inputs and outputs. ANN have more advantages that can approximate to model both linear and nonlinear structures in time series, they are not able to handling both structures equally well. The autoregressive integrated moving average (ARIMA) model and two ANN models namely, Radial basis function neural networks (RBFNN), and Elman recurrent neural networks (ERNN) methods were applied to Hyderabad airport traffic data. The data obtained for 15 years from 2002–2003 to 2016–2017 about domestic and international passenger of International Airport of Hyderabad, India. In this research paper, we compared the performances of ARIMA, RBFNN and ERNN were based on three measures: mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE). The results showed that RBFNN obtained the smallest MAE, MAPE and RMSE in both the modeling and forecasting processes. The performances of the three models ranked in ascending order were: ARIMA, ERNN and the RBFNN model. Keywords: T ime series, forecasting, artificial neural networks, ARIMA models, radial basis function neural networks, and Elman recurrent neural networks Cite this Article R. Ramakrishna, Berhe Aregay, Tewodros Gebregergs. The Comparison in Time Series Forecasting of Air Traffic Data by Autoregressive Integrated Moving Average Model, Radial Basis Function and Elman Recurrent Neural Networks. Research & Reviews: Journal of Statistics . 2018; 7(3): 75–90p.

One of the difficulties encountered in the application of reinforcement learning methods to real-world problem is the generalization of large-scale or continuous state space. In order to solve the curse of dimensionality problem caused by discretizing continuous state space, a kind of Q-learning method for continuous state space based on a dynamic Elman neural network was proposed in this paper. The inputs and the output of Elman network are the system state-action pair and the corresponding Q-value. That is, Elman network is used to estimate the Q-value of state-action pair on-line. Eligibility trace for connecting weights is introduced by borrowing ideas from the eligibility trace mechanism of state in temporal difference algorithm to improve the learning speed of neural network. Computer simulations on mountain car control illustrate the performance and applicability of the proposed Q-learning scheme.

We propose to use segment graph convolutional and recurrent neural networks (Seg-GCRNs), which use only word embedding and sentence syntactic dependencies, to classify relations from clinical notes without manual feature engineering. In this study, the relations between 2 medical concepts are classified by simultaneously learning representations of text segments in the context of sentence syntactic dependency: preceding, concept1, middle, concept2, and succeeding segments. Seg-GCRN was systematically evaluated on the i2b2/VA relation classification challenge datasets. Experiments show that Seg-GCRN attains state-of-the-art micro-averaged F-measure for all 3 relation categories: 0.692 for classifying medical treatment-problem relations, 0.827 for medical test-problem relations, and 0.741 for medical problem-medical problem relations. Comparison with the previous state-of-the-art segment convolutional neural network (Seg-CNN) suggests that adding syntactic dependency information helps refine medical word embedding and improves concept relation classification without manual feature engineering. Seg-GCRN can be trained efficiently for the i2b2/VA dataset on a GPU platform.

In recent years, electroencephalogram (EEG) e-motion recognition has been becoming an emerging field in artificial intelligence area, which can reflect the relation between emotional states and brain activity. In this paper, we designed a novel architecture, i.e., broad dynamical graph learning system (BDGLS), to deal with EEG signals. By integrating the advantage of dynamical graph convolution neural networks (DGCNN) and broad learning system (BLS), BDGLS has the ability of extracting features on non-Euclidean domain and randomly generating nodes to find the desired connection weights simultaneously. We evaluated our system on SJTU emotion EEG dataset (SEED), and used differential entropy (DE) features as input data. In the experiments, BDGLS achieved the best result, compared with the state-of-the-art methods, e.g., support vector machine (SVM), deep belief networks (DBN), graph convolutional neural networks (DCNN) and DGCNN. Especially the performance on all-frequency band of DE features, BDGLS reached the highest average recognition accuracy of 93.66% with the standard deviation of 6.11%. The result demonstrated the excellent classification ability of BDGLS in EEG emotion recognition.

Electricity plays a very important role in daily modern human-life activities. An electricity company must always guarantee the continuity and adequate supply to its customers. Consequently, it must always be able to predict the future electricity demand to be supplied by considering various influencing factors. Many forecasting methods have been investigated and proposed by researchers to help in predicting the future electricity demand to be fulfilled, which is a paramount information in planning the transmission and distribution infrastructure and the generation plants to be built. In this study, two forecasting methods are described, explored and compared to provide alternative consideration in choosing the method. An artificial intelligence-based forecasting method, the Recurrent Neural Network (RNN), is to be compared to a conventional forecasting method, the Vector Autoregressive (VAR). The comparison is based on the parameters of Root Mean Square Error (RMSE) and the Mean Absolute Error (MAE). Both methods are implemented to predict the shortterm electricity load demand in Malang City, the second largest city after Surabaya in East Java province of Indonesia. The existing load data have been obtained from local electricity company, whereas the weather data have been taken from the Meteoblue Climatology NOAA. The architecture modelling of the RNN and VAR methods are performed in such a way to produce an accurate forecasting result. Based on the RMSE and MAE values, the prediction results of short-term electricity load in Malang city using the RNN method with hidden neuron variations indicate the lower values of RMSE and MAE, indicating better accuracy and performance, than the use of the VAR method with lag value variation.