HiSTGNN: Hierarchical spatio-temporal graph neural network for weather forecasting

Non-point source pollution from agricultural activities poses a significant threat to water quality by introducing excess nutrients like nitrogen into aquatic ecosystems, leading to issues such as eutrophication and groundwater contamination. In agricultural watersheds, nitrate transport involves intricate physical, chemical, and biological processes influenced by meteorological conditions, hydrological features, and spatial topologies, making accurate short-term predictions challenging. Traditional data-driven deep learning models often fail to incorporate physical constraints and complex spatiotemporal dynamics, limiting their interpretability and predictive accuracy. Here we show a hierarchical transformer and graph neural network model that accurately predicts watershed nitrate concentrations by integrating multi-source data and simulating pollutant migration. The model captures nonlinear multivariate temporal patterns through hierarchical transformers, fuses global meteorological and local hydrological features via neural networks, and models runoff topologies with physically constrained graph neural networks. For predicting the concentration changes of pollutants discharged from watersheds, it outperforms baselines like multi-layer perceptrons, recurrent neural networks, and long short-term memory networks, with state-of-the-art performance in root mean square error, mean absolute error, and R2. Ablation studies confirm the essential roles of multi-source data integration and watershed topological modeling in enhancing performance. This method of directly modeling physical processes by leveraging the characteristics of different neural network architectures opens up a new path for addressing the interpretability problem in neural earth system modeling, apart from the process-guided deep learning and differentiable modelling methods.

ABSTRACTA cognitive map is an internal model of the external world and contains the spatial representation of the surrounding environment. The existence of the cognitive map was first identified in rats; rats can navigate to their desired destination using cognitive maps while dealing with environmental uncertainty. We performed a mobile robot navigation experiment where obstacles were randomly placed using hierarchical recurrent neural network (HRNN) with multiple timescales. The HRNN was trained to navigate the mobile robot to the destination indicated by a snapshot image. After the training, the HRNN was able to successfully avoid the obstacles and navigate to the destination from any location in the environment. Analysis of the internal states of the HRNN showed that the module with fast timescale handles obstacle avoidance and the one with slow timescale has spatial representation corresponding to the spatial position of the destination. Moreover, in the experiment wherein the novel path appeared, the trained HRNN performed shortcut behavior. The shortcut behavior shows that the HRNN performed navigation using the self-organized spatial representation in the slow recurrent neural network. This indicates that training of goal-oriented navigation, i.e. the navigation motivated by a snapshot image of the destination results in the self-organization of cognitive map-like representation.

This paper presents a waveform modeling and generation method using hierarchical recurrent neural networks (HRNN) for speech bandwidth extension (BWE). Different from conventional BWE methods that predict spectral parameters for reconstructing wideband speech waveforms, this BWE method models and predicts waveform samples directly without using vocoders. Inspired by SampleRNN, which is an unconditional neural audio generator, the HRNN model represents the distribution of each wideband or high-frequency waveform sample conditioned on the input narrowband waveform samples using a neural network composed of long short-term memory (LSTM) layers and feed-forward layers. The LSTM layers form a hierarchical structure and each layer operates at a specific temporal resolution to efficiently capture long-span dependencies between temporal sequences. Furthermore, additional conditions, such as the bottleneck features derived from narrowband speech using a deep neural network based state classifier, are employed as auxiliary input to further improve the quality of generated wideband speech. The experimental results of comparing several waveform modeling methods show that the HRNN-based method can achieve better speech quality and run-time efficiency than the dilated convolutional neural network based method and the plain sample-level recurrent neural network based method. Our proposed method also outperforms the conventional vocoder-based BWE method using LSTM-RNNs in terms of the subjective quality of the reconstructed wideband speech.

Image representation as graphs can enhance the understanding of image semantics and facilitate multi-scale image representation. However, existing methods often overlook the significance of the relationship between elements at each scale or fail to encode the hierarchical relationship between graph elements. To cope with that, we introduce four novel approaches for graph construction from images. These approaches utilize hierarchical image segmentation techniques to generate segmentations at multiple scales, and one of them incorporates edges to encode the relationships at each scale. Leveraging these representations, we present two new models: the Hierarchical Graph Convolutional Network for Image Classification (HGCIC) and the Hierarchical Image Graph with Scale Importance (HIGSI). HGCIC uses an adaptive depth to capture significant features and patterns at different scales, while HIGSI employs a novel readout function that weighs the importance of each scale when generating a fixed-size graph representation. Experimental results with CIFAR-10 and STL-10 datasets show that the HIGSI model outperforms (or closely matches) state-of-the-art models. The model also utilizes smaller graphs, reaching the point of using graphs with 50% of the number of nodes compared to other approaches. Additionally, HIGSI outperforms models trained with only the base graph used to create the hierarchy, achieving up to 11.54% better performance while using fewer parameters.

The Sixth International Symposium on Neural Networks (ISNN 2009)

The human nervous system is an ensemble of connected neuronal networks. Modeling and system identification of the human nervous system helps us understand how the brain processes sensory input and controls responses at the systems level. This study aims to propose an advanced approach based on a hierarchical neural network and non-linear system identification method to model neural activity in the nervous system in response to an external somatosensory input. The proposed approach incorporates basic concepts of Non-linear AutoRegressive Moving Average Model with eXogenous input (NARMAX) and neural network to acknowledge non-linear closed-loop neural interactions. Different from the commonly used polynomial NARMAX method, the proposed approach replaced the polynomial non-linear terms with a hierarchical neural network. The hierarchical neural network is built based on known neuroanatomical connections and corresponding transmission delays in neural pathways. The proposed method is applied to an experimental dataset, where cortical activities from ten young able-bodied individuals are extracted from electroencephalographic signals while applying mechanical perturbations to their wrist joint. The results yielded by the proposed method were compared with those obtained by the polynomial NARMAX and Volterra methods, evaluated by the variance accounted for (VAF). Both the proposed and polynomial NARMAX methods yielded much better modeling results than the Volterra model. Furthermore, the proposed method modeled cortical responded with a mean VAF of 69.35% for a three-step ahead prediction, which is significantly better than the VAF from a polynomial NARMAX model (mean VAF 47.09%). This study provides a novel approach for precise modeling of cortical responses to sensory input. The results indicate that the incorporation of knowledge of neuroanatomical connections in building a realistic model greatly improves the performance of system identification of the human nervous system.

Inspired by recent improvements in point cloud processing for autonomous navigation, we focus on using hierarchical graph neural networks for processing and feature learning over large-scale outdoor LiDAR point clouds. We observe that existing GNN based methods fail to overcome challenges of scale and irregularity of points in outdoor datasets. Addressing the need to preserve structural details while learning over a larger volume efficiently, we propose Hierarchical Point Graph Neural Network (HPGNN). It learns node features at various levels of graph coarseness to extract information. This enables to learn over a large point cloud while retaining fine details that existing point-level graph networks struggle to achieve. Connections between multiple levels enable a point to learn features in multiple scales, in a few iterations. We design HPGNN as a purely GNN-based approach, so that it offers modular expandability as seen with other point-based and Graph network baselines. To illustrate the improved processing capability, we compare previous point based and GNN models for semantic segmentation with our HPGNN, achieving a significant improvement for GNNs (+36.7 mIoU) on the SemanticKITTI dataset.

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

The automatic classification of electrocardiograms (ECGs) plays a crucial role in the early diagnosis of cardiovascular diseases. In recent research, deep neural network (DNN)-based methods have garnered significant attention due to their exceptional feature extraction capabilities. However, these methods face challenges in dealing with the complex inter- and intra-class dependencies inherent in different arrhythmias. We propose a classification model for electrocardiograms based on the Graph Convolutional Neural Network (EGCNet) to address this. In our EGCNet framework, we first employ the row and column 1-dimensional residual convolution to learn intra-class dependency. Then, we construct a hierarchical graph structure to model the inter-class dependency. In such a graph, each node represents one type of disease, and the edges describe the relationship between diseases. To reduce the complexity of inter-class dependency, we build a hierarchical graph structure by treating similar nodes as the same type. Thus, we can learn the discriminant dependency in the higher-level graph space and feed this high-level knowledge back to the lower-level graph space. Finally, we evaluated our method using publicly available datasets from the PhysioNet/Computing Challenge 2020 ECG data. For instance, our model achieved robust performance in the 24-label experiment with a precision of 0.753 and an F1 score of 0.722.

This paper proposes a hybrid neuro-evolutive algorithm (NEA) that uses a compact indirect encoding scheme (IES) for representing its genotypes, moreover has the ability to reuse the genotypes and automatically build modular, hierarchical and recurrent neural networks. A genetic algorithm (GA) evolves a Lindenmayer System that is used to design the neural network's architecture. This basic neural codification confers scalability and search space reduction in relation to other methods. Furthermore, the system uses a parallel genome scan engine that increases both the implicit parallelism and convergence of the GA. The fitness function of the NEA rewards economical artificial neural networks (ANNs) that are easily implemented. The NEA was tested on five real-world classification datasets and three well-known datasets for time series forecasting (TSF). The results are statistically compared against established state-of-the-art algorithms and various forecasting methods (ADANN, ARIMA, UCM, and Forecast Pro). In most cases, our NEA outperformed the other methods, delivering the most accurate classification and time series forecasting with the least computational effort. These superior results are attributed to the improved effectiveness and efficiency of NEA in the decision-making process.

This paper proposes noisy speech recognition using hierarchical singleton-type recurrent neural fuzzy networks (HSRNFNs). The proposed HSRNFN is a hierarchical connection of two singleton-type recurrent neural fuzzy networks (SRNFNs), where one is used for noise filtering and the other for recognition. The SRNFN is constructed by recurrent fuzzy if-then rules with fuzzy singletons in the consequences, and their recurrent properties make them suitable for processing speech patterns with temporal characteristics. In n words recognition, n SRNFNs are created for modeling n words, where each SRNFN receives the current frame feature and predicts the next one of its modeling word. The prediction error of each SRNFN is used as recognition criterion. In filtering, one SRNFN is created, and each SRNFN recognizer is connected to the same SRNFN filter, which filters noisy speech patterns in the feature domain before feeding them to the SRNFN recognizer. Experiments with Mandarin word recognition under different types of noise are performed. Other recognizers, including multilayer perceptron (MLP), time-delay neural networks (TDNNs), and hidden Markov models (HMMs), are also tested and compared. These experiments and comparisons demonstrate good results with HSRNFN for noisy speech recognition tasks.

Speaker change detection (SCD) is an important task in dialog modeling. Our paper addresses the problem of text-based SCD, which differs from existing audio-based studies and is useful in various scenarios, for example, processing dialog transcripts where speaker identities are missing (e.g., OpenSubtitle), and enhancing audio SCD with textual information. We formulate text-based SCD as a matching problem of utterances before and after a certain decision point; we propose a hierarchical recurrent neural network (RNN) with static sentence-level attention. Experimental results show that neural networks consistently achieve better performance than feature-based approaches, and that our attention-based model significantly outperforms non-attention neural networks.

Hierarchical Neural Networks [e.g., deep neural networks (DNNs)] have recently gained increasing attention for image classification task. Most of the previous works devote to stack multiple levels of feature learning modules with the hope that higher level modules can represent more abstract semantics of the image data. However, training of such hierarchical network is typically cast as a time-consuming non-convex optimization problem, and its effectiveness for image feature representation critically depends on expertise in parameter tuning with various ad hoc tricks. To address these issues, we advocate a biologically-inspired hierarchical neural network. One key philosophy is that the higher-level modules in the network should correct misclassification induced by the lower ones. Given this idea, a sequential stacking strategy of basic feature learning modules is presented. In practical, an efficient layer wise learning and decision aggregation method is applied to boost the hierarchical network, which is different from naive layers cascading and end-to-end finetuning, and we term it as AdaboostNet. Moreover, the basic feature learning module is set as extreme learning machine (ELM), an effective and cheaply-optimized model. Experiments are conducted on benchmark datasets to evaluate our claims. It shows that the proposed network achieves comparable or better image classification accuracy and training efficiency than traditional DNNs and hierarchical ELMs.

A hierarchical neural network which can recognize handwritten Tamil characters independently of their position and size is described. A character is input as a pattern of (x,y) coordinates, subjected to very simple preprocessing. The network is trained by using the backpropagation learning algorithm. The performance of the hierarchical neural network is compared with and found superior to the performance of two other classifiers, namely, a single neural network approach and the method of moments in conjunction with a feedforward network. >

Recurrent neural networks (RNNs) can be used to handle sequential patterns and have been used for speech recognition. To overcome the shortcomings of RNN, recurrent sub neural networks (RSNNs) are used, where an RSNN is built independently for each class. The training algorithm of the RSNN is based on the backpropagation algorithm. Speaker dependent connected Chinese digit-speech recognition experiments were carried out. Some factors influencing the performance of RSNNs have been studied. The experiments show that RSNN is easier to train and gives higher performance than RNN.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>