A Multi-Stage Hierarchical Relational Graph Neural Network for Multimodal Sentiment Analysis

Image representation as graphs can enhance the understanding of image semantics and facilitate multi-scale image representation. However, existing methods often overlook the significance of the relationship between elements at each scale or fail to encode the hierarchical relationship between graph elements. To cope with that, we introduce four novel approaches for graph construction from images. These approaches utilize hierarchical image segmentation techniques to generate segmentations at multiple scales, and one of them incorporates edges to encode the relationships at each scale. Leveraging these representations, we present two new models: the Hierarchical Graph Convolutional Network for Image Classification (HGCIC) and the Hierarchical Image Graph with Scale Importance (HIGSI). HGCIC uses an adaptive depth to capture significant features and patterns at different scales, while HIGSI employs a novel readout function that weighs the importance of each scale when generating a fixed-size graph representation. Experimental results with CIFAR-10 and STL-10 datasets show that the HIGSI model outperforms (or closely matches) state-of-the-art models. The model also utilizes smaller graphs, reaching the point of using graphs with 50% of the number of nodes compared to other approaches. Additionally, HIGSI outperforms models trained with only the base graph used to create the hierarchy, achieving up to 11.54% better performance while using fewer parameters.

HiSTGNN: Hierarchical spatio-temporal graph neural network for weather forecasting

Non-point source pollution from agricultural activities poses a significant threat to water quality by introducing excess nutrients like nitrogen into aquatic ecosystems, leading to issues such as eutrophication and groundwater contamination. In agricultural watersheds, nitrate transport involves intricate physical, chemical, and biological processes influenced by meteorological conditions, hydrological features, and spatial topologies, making accurate short-term predictions challenging. Traditional data-driven deep learning models often fail to incorporate physical constraints and complex spatiotemporal dynamics, limiting their interpretability and predictive accuracy. Here we show a hierarchical transformer and graph neural network model that accurately predicts watershed nitrate concentrations by integrating multi-source data and simulating pollutant migration. The model captures nonlinear multivariate temporal patterns through hierarchical transformers, fuses global meteorological and local hydrological features via neural networks, and models runoff topologies with physically constrained graph neural networks. For predicting the concentration changes of pollutants discharged from watersheds, it outperforms baselines like multi-layer perceptrons, recurrent neural networks, and long short-term memory networks, with state-of-the-art performance in root mean square error, mean absolute error, and R2. Ablation studies confirm the essential roles of multi-source data integration and watershed topological modeling in enhancing performance. This method of directly modeling physical processes by leveraging the characteristics of different neural network architectures opens up a new path for addressing the interpretability problem in neural earth system modeling, apart from the process-guided deep learning and differentiable modelling methods.

Inspired by recent improvements in point cloud processing for autonomous navigation, we focus on using hierarchical graph neural networks for processing and feature learning over large-scale outdoor LiDAR point clouds. We observe that existing GNN based methods fail to overcome challenges of scale and irregularity of points in outdoor datasets. Addressing the need to preserve structural details while learning over a larger volume efficiently, we propose Hierarchical Point Graph Neural Network (HPGNN). It learns node features at various levels of graph coarseness to extract information. This enables to learn over a large point cloud while retaining fine details that existing point-level graph networks struggle to achieve. Connections between multiple levels enable a point to learn features in multiple scales, in a few iterations. We design HPGNN as a purely GNN-based approach, so that it offers modular expandability as seen with other point-based and Graph network baselines. To illustrate the improved processing capability, we compare previous point based and GNN models for semantic segmentation with our HPGNN, achieving a significant improvement for GNNs (+36.7 mIoU) on the SemanticKITTI dataset.

Graph Neural Networks (GNNs) have become a prominent approach to machine learning with graphs and have been increasingly applied in a multitude of domains. Nevertheless, since most existing GNN models are based on flat message-passing mechanisms, two limitations need to be tackled: (i) they are costly in encoding long-range information spanning the graph structure; (ii) they are failing to encode features in the high-order neighbourhood in the graphs as they only perform information aggregation across the observed edges in the original graph. To deal with these two issues, we propose a novel Hierarchical Message-passing Graph Neural Networks framework. The key idea is generating a hierarchical structure that re-organises all nodes in a flat graph into multi-level super graphs, along with innovative intra- and inter-level propagation manners. The derived hierarchy creates shortcuts connecting far-away nodes so that informative long-range interactions can be efficiently accessed via message passing and incorporates meso- and macro-level semantics into the learned node representations. We present the first model to implement this framework, termed Hierarchical Community-aware Graph Neural Network (HC-GNN), with the assistance of a hierarchical community detection algorithm. The theoretical analysis illustrates HC-GNN’s remarkable capacity in capturing long-range information without introducing heavy additional computation complexity. Empirical experiments conducted on 9 datasets under transductive, inductive, and few-shot settings exhibit that HC-GNN can outperform state-of-the-art GNN models in network analysis tasks, including node classification, link prediction, and community detection. Moreover, the model analysis further demonstrates HC-GNN’s robustness facing graph sparsity and the flexibility in incorporating different GNN encoders.

Hierarchical graph representations in digital pathology.

Graph neural network (GNN) is popular now to solve the tasks in non-Euclidean space and most of them learn deep embeddings by aggregating the neighboring nodes. However, these methods are prone to some problems such as over-smoothing because of the single-scale perspective field and the nature of low-pass filter. To address these limitations, we introduce diffusion scattering network (DSN) to exploit high-order patterns. With observing the complementary relationship between multi-layer GNN and DSN, we propose Hierarchical Diffusion Scattering Graph Neural Network (HDS-GNN) to efficiently bridge DSN and GNN layer by layer to supplement GNN with multi-scale information and band-pass signals. Our model extracts node-level scattering representations by intercepting the low-pass filtering, and adaptively tunes the different scales to regularize multi-scale information. Then we apply hierarchical representation enhancement to improve GNN with the scattering features. We benchmark our model on nine real-world networks on the transductive semi-supervised node classification task. The experimental results demonstrate the effectiveness of our method.

Modeling the evolution process of industrial cascading faults by capturing causal relationships has achieved widespread attention in the attack detection and security control field. However, the highly coupled nature of numerous sensors and control loops in complex systems makes it difficult for existing root cause identification (RCI) methods to accurately and clearly characterize the fault-induced causal relationships. To address this challenge, this article proposes the hierarchical causal graph neural networks (HCGNNs) to reveal the root cause in an end-to-end manner with the aid of cascading failure analysis. First, the cause–effect weighted adjacency matrix is designed to expand the connections between nodes from correlations to causalities in a graph neural network (GNN). On this basis, the proposed hierarchical network structure can collaboratively learn the intraunit and interunit causalities, which significantly reduces the spurious and redundant causal relations while improving the modeling efficiency. Additionally, a new cascading failure analysis method is formulated to quantify the root cause and clarify propagation paths from a more objective perspective. Finally, the effectiveness and superiority of the proposed method are verified by the Tennessee Eastman platform and the real-world coal mill group.

The objective of cross-scene hyperspectral image (HSI) classification is to develop models capable of adapting to the “domain gap” that exists between different scenes, enabling accurate object classification in previously unseen scenes. Many researchers have devised various domain adaptation techniques aimed at aligning the statistical or spectral distributions of data from diverse scenes. However, many previous studies have overlooked the potential benefits of incorporating spatial topological information from hyperspectral imagery, which could provide a more accurate representation of the inherent data structure in HSIs. To overcome this issue, we introduce an innovative approach for cross-scene HSI classification, founded on hierarchical prototype graph alignment. Specifically, this method leverages prototypes as representative embedded representations of all samples within the same class. By employing multiple graph convolution and pooling operations, multi-scale domain alignment is attained. Beyond statistical distribution alignment, we integrate graph matching to effectively reconcile semantic and topological information. Experimental results on several datasets achieve significantly improved accuracy and generalization capabilities for cross-scene HSI classification tasks.

The human nervous system is an ensemble of connected neuronal networks. Modeling and system identification of the human nervous system helps us understand how the brain processes sensory input and controls responses at the systems level. This study aims to propose an advanced approach based on a hierarchical neural network and non-linear system identification method to model neural activity in the nervous system in response to an external somatosensory input. The proposed approach incorporates basic concepts of Non-linear AutoRegressive Moving Average Model with eXogenous input (NARMAX) and neural network to acknowledge non-linear closed-loop neural interactions. Different from the commonly used polynomial NARMAX method, the proposed approach replaced the polynomial non-linear terms with a hierarchical neural network. The hierarchical neural network is built based on known neuroanatomical connections and corresponding transmission delays in neural pathways. The proposed method is applied to an experimental dataset, where cortical activities from ten young able-bodied individuals are extracted from electroencephalographic signals while applying mechanical perturbations to their wrist joint. The results yielded by the proposed method were compared with those obtained by the polynomial NARMAX and Volterra methods, evaluated by the variance accounted for (VAF). Both the proposed and polynomial NARMAX methods yielded much better modeling results than the Volterra model. Furthermore, the proposed method modeled cortical responded with a mean VAF of 69.35% for a three-step ahead prediction, which is significantly better than the VAF from a polynomial NARMAX model (mean VAF 47.09%). This study provides a novel approach for precise modeling of cortical responses to sensory input. The results indicate that the incorporation of knowledge of neuroanatomical connections in building a realistic model greatly improves the performance of system identification of the human nervous system.

In recent years, graph neural network is used to process graph data and has been successfully applied to graph node classification task. Due to the complexity of graph structure and the difficulty of obtaining node labels, node classification in datasets with fewer labels becomes a challenge. Existing node classification problems with few-label nodes datasets usually use neighbor aggregation schemes. However, these methods lack the “graph pooling mechanism,” which makes these models impossible to make full use of the global graph information. To solve the problem, we propose a Hierarchical Structure-Feature aware Graph Neural Network (HSFGNN) model. The model learns node features by a hierarchical mechanism that repeatedly utilizes coarsening and refining methods at different levels. Firstly, we use topological information, node characteristics and connectivity of the graph to construct an intuitive and effective coarsening part based on multivariable analysis technology. Secondly, we aggregate the unselected nodes’ features into the top-k nodes, so that the coarser nodes contain more abundant valid graph information. Finally, we propose an additional attention mechanism to obtain more accurate final representations of nodes. Experimental results show that the HSFGNN model achieves outstanding classification accuracy on the dataset with less labeled data.

Recent years have witnessed the rapidly growing of the amount of semistructured documents in real-world applications. Due to the huge size of the real-world data, how to manage semistructured documents effectively is a big challenge for researchers. As a fundamental task in natural language processing field, document classification is a feasible way to handle the large-scale semistructured documents. However, existing methods fail to explicitly take advantage of the hierarchical semantics in semistructured documents. It's known that the contained semantics is beneficial for understanding the semistructured documents. Considering the hierarchical structure of a given semistructured document, we propose a semistructured document classification framework which explicitly utilizes the semantic hierarchical attention mechanism. More specifically, the hierarchical attention mechanism and graph neural network are employed to model semistructured documents, by which the multilevel semantic relationships and grammatical information are considered. Moreover, we propose an adaptive class cost learning method to treat the issue of data imbalance. Comprehensive experiments are conducted on two real-world data sets, and the results demonstrate that our framework performs better than selected baselines for semistructured document classification.

Modern mechanical equipment increasingly relies on multi-sensor data for health monitoring, which generates complex, heterogeneous, non-Euclidean data that conventional methods struggle to process. While graph neural networks (GNNs) are promising for such data, existing GNN-based fault diagnosis methods often suffer from two key limitations: 1) neglecting morphological details when mapping time-series data to graph structures, resulting in the loss of critical information; and 2) The lack of domain knowledge regarding fault propagation to guide the graph construction process makes it difficult to capture complex correlations among heterogeneous sensor data. To overcome these challenges, this paper proposes a hierarchical graph neural network (HGNN) for the dynamic fusion of heterogeneous data. Specifically, our method first builds sensor-level graphs by extracting discriminative shapelets and applying a pattern-distance-based <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i>-nearest neighbor algorithm to capture fine-grained local features. Subsequently, these individual graphs are dynamically interconnected into a comprehensive system-level graph using a dual-metric key-node screening method and a novel Feature-Type Dynamic Weighted Similarity (FTDWS) metric. This hierarchical approach preserves critical local fault signatures while effectively modeling system-wide fault propagation and sensor correlations across heterogeneous sensors. Finally, a graph attention network performs classification on this system-level representation. Extensive experiments on two public datasets and one real-world industrial coal mill dataset demonstrate the effectiveness and superiority of our proposed method.

Most sequential recommendation (SR) systems employing graph neural networks (GNNs) only model a user's interaction sequence as a flat graph without hierarchy, overlooking diverse factors in the user's preference. Moreover, the timespan between interacted items is not sufficiently utilized by previous models, restricting SR performance gains. To address these problems, we propose a novel SR system employing a hierarchical graph neural network (HGNN) to model factorial user preferences. Specifically, a timespan-aware sequence graph (TSG) for the target user is first constructed with the timespan among interacted items. Next, all original nodes in TSG are softly clustered into factor nodes, each of which represents a certain factor of the user's preference. At last, all factor nodes' representations are used together to predict SR results. Our extensive experiments upon two datasets justify that our HGNN-based factorial user modeling obtains better SR performance than the state-of-the-art SR models.

Implicit emotions are often expressed through implicit and weak clues between modalities due to the lack of explicit emotional feature words, representing a significant challenge for multimodal sentiment analysis. In order to improve implicit emotion recognition, this paper proposes a multimodal sentiment analysis method that integrates KAN and the modal dynamic fusion mechanism. This method first introduces the KAN structure to construct a modal feature encoder to enhance the emotional expression ability of features. Then, the emotional contribution weight of each modality is calculated using the difference between the unimodal and multimodal sentiment scores, and the cross-attention mechanism guided by the main modality is used for feature fusion. Experiments on four datasets, CH-SIMS, CH-SIMSv2, MOSI, and MOSEI, show that the proposed method significantly outperforms the mainstream model in multiple indicators, especially when dealing with samples with implicit or ambiguous emotional expressions. The results verify the effectiveness of enhancing feature encoding capabilities and utilizing modal asymmetry information in implicit sentiment analysis.