Hierarchical Graph Neural Networks Based on Multi-Scale Image Representations

Inspired by recent improvements in point cloud processing for autonomous navigation, we focus on using hierarchical graph neural networks for processing and feature learning over large-scale outdoor LiDAR point clouds. We observe that existing GNN based methods fail to overcome challenges of scale and irregularity of points in outdoor datasets. Addressing the need to preserve structural details while learning over a larger volume efficiently, we propose Hierarchical Point Graph Neural Network (HPGNN). It learns node features at various levels of graph coarseness to extract information. This enables to learn over a large point cloud while retaining fine details that existing point-level graph networks struggle to achieve. Connections between multiple levels enable a point to learn features in multiple scales, in a few iterations. We design HPGNN as a purely GNN-based approach, so that it offers modular expandability as seen with other point-based and Graph network baselines. To illustrate the improved processing capability, we compare previous point based and GNN models for semantic segmentation with our HPGNN, achieving a significant improvement for GNNs (+36.7 mIoU) on the SemanticKITTI dataset.

HiSTGNN: Hierarchical spatio-temporal graph neural network for weather forecasting

The objective of cross-scene hyperspectral image (HSI) classification is to develop models capable of adapting to the “domain gap” that exists between different scenes, enabling accurate object classification in previously unseen scenes. Many researchers have devised various domain adaptation techniques aimed at aligning the statistical or spectral distributions of data from diverse scenes. However, many previous studies have overlooked the potential benefits of incorporating spatial topological information from hyperspectral imagery, which could provide a more accurate representation of the inherent data structure in HSIs. To overcome this issue, we introduce an innovative approach for cross-scene HSI classification, founded on hierarchical prototype graph alignment. Specifically, this method leverages prototypes as representative embedded representations of all samples within the same class. By employing multiple graph convolution and pooling operations, multi-scale domain alignment is attained. Beyond statistical distribution alignment, we integrate graph matching to effectively reconcile semantic and topological information. Experimental results on several datasets achieve significantly improved accuracy and generalization capabilities for cross-scene HSI classification tasks.

Non-point source pollution from agricultural activities poses a significant threat to water quality by introducing excess nutrients like nitrogen into aquatic ecosystems, leading to issues such as eutrophication and groundwater contamination. In agricultural watersheds, nitrate transport involves intricate physical, chemical, and biological processes influenced by meteorological conditions, hydrological features, and spatial topologies, making accurate short-term predictions challenging. Traditional data-driven deep learning models often fail to incorporate physical constraints and complex spatiotemporal dynamics, limiting their interpretability and predictive accuracy. Here we show a hierarchical transformer and graph neural network model that accurately predicts watershed nitrate concentrations by integrating multi-source data and simulating pollutant migration. The model captures nonlinear multivariate temporal patterns through hierarchical transformers, fuses global meteorological and local hydrological features via neural networks, and models runoff topologies with physically constrained graph neural networks. For predicting the concentration changes of pollutants discharged from watersheds, it outperforms baselines like multi-layer perceptrons, recurrent neural networks, and long short-term memory networks, with state-of-the-art performance in root mean square error, mean absolute error, and R2. Ablation studies confirm the essential roles of multi-source data integration and watershed topological modeling in enhancing performance. This method of directly modeling physical processes by leveraging the characteristics of different neural network architectures opens up a new path for addressing the interpretability problem in neural earth system modeling, apart from the process-guided deep learning and differentiable modelling methods.

Scene graph generation denotes the process of parsing a visual input into a graph representation for downstream reasoning tasks. Most existing research models input scenes as flat scene graphs, capturing solely on horizontal relationships between two objects, while disregarding vertical dependencies. Such flat scene graphs merely describe direct relationships between objects, and cannot represent higher-level semantics. The lack of hierarchical relationships also limits the performance of scene graphs in downstream reasoning tasks. In this work, we propose hierarchical scene graph generation, a new problem task that requires the model to generate scene graphs with clear hierarchy. To achieve this goal, a hierarchical scene graph generation with coarse-to-fine reasoning framework is proposed. It contains three stages: first, summarize the main caption of the scene; second, describe the main visual elements related to the theme; and finally, add secondary elements with background. To facilitate training, a high-quality hierarchical scene graph dataset that comprises 40 K image-scene description pairs is constructed. Utilize rich knowledge and powerful multi-turn conversation capabilities of multi-modal large language models, the proposed framework can not only achieve better semantic understanding and relational modeling, but also seamlessly integrating its relational modeling capabilities to enhance visual-language tasks. Extensive experimental results demonstrate that our method constructs an effective scene representation, outperforming the state-of-the-art by 3.35% in recall while generating significantly fewer average triplets 8.7vs. 9.2. The results on downstream tasks also indicate that hierarchical scene graph significantly contributes to visual-language interactive reasoning.

Multimodal sentiment analysis targets at accurately perceiving the emotional states by incorporating related information from multiple sources. However, existing methods mostly neglect the unbalanced contributions and inherent relational interactions across distinct modalities. In this paper, we propose a multi-stage hierarchical relational graph neural network (MHRG), catering to intra- and inter-modal dynamics learning with modality calibration. In the first stage, modality-specific graph convolution modules are introduced to learn the intra-modal sequential semantics. In the second, we design a modality-adaptive modification module to determine the contribution of each modality based on the prediction confidence. Finally, diverse inter-modal dynamics are considered respectively by a novel hierarchical relational graph fusion method for further aggregation according to the type of interactions. Extensive experiments on benchmark datasets demonstrate that MHRG outperforms the existing methods and achieves the state-of-the-art performance.

Graph Neural Networks (GNNs) have become a prominent approach to machine learning with graphs and have been increasingly applied in a multitude of domains. Nevertheless, since most existing GNN models are based on flat message-passing mechanisms, two limitations need to be tackled: (i) they are costly in encoding long-range information spanning the graph structure; (ii) they are failing to encode features in the high-order neighbourhood in the graphs as they only perform information aggregation across the observed edges in the original graph. To deal with these two issues, we propose a novel Hierarchical Message-passing Graph Neural Networks framework. The key idea is generating a hierarchical structure that re-organises all nodes in a flat graph into multi-level super graphs, along with innovative intra- and inter-level propagation manners. The derived hierarchy creates shortcuts connecting far-away nodes so that informative long-range interactions can be efficiently accessed via message passing and incorporates meso- and macro-level semantics into the learned node representations. We present the first model to implement this framework, termed Hierarchical Community-aware Graph Neural Network (HC-GNN), with the assistance of a hierarchical community detection algorithm. The theoretical analysis illustrates HC-GNN’s remarkable capacity in capturing long-range information without introducing heavy additional computation complexity. Empirical experiments conducted on 9 datasets under transductive, inductive, and few-shot settings exhibit that HC-GNN can outperform state-of-the-art GNN models in network analysis tasks, including node classification, link prediction, and community detection. Moreover, the model analysis further demonstrates HC-GNN’s robustness facing graph sparsity and the flexibility in incorporating different GNN encoders.

In this paper, we present an edge detection method based on the thin-plate spline with tension. Under regularization theory, the image is represented in a convolution form between the original image data and a two-dimensional kernel. This convolution kernel is derived from a PDE, which is related to a second order (thin-plate or bending) term and a first order (membrane or tension) term. This image representation involves two parameters: a smoothing parameter (or scale parameter) and a weighted smoothness parameter (which controls the degrees of the continuity of the reconstruction by placing a different weighting on the thin-plate and tension terms). By tuning these parameters, the image can be represented at different scales and with different smoothness requirements. Based on this convolution representation, the edges can be detected by differentiating the image to look for the zero-crossings of the Laplacian. By tuning the values of these two parameters, the edges at the different scales will be extracted.KeywordsEdge DetectionCoarse ScaleModify Bessel FunctionNoise SuppressionConvolution KernelThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Hierarchical graph representations in digital pathology.

The automatic classification of electrocardiograms (ECGs) plays a crucial role in the early diagnosis of cardiovascular diseases. In recent research, deep neural network (DNN)-based methods have garnered significant attention due to their exceptional feature extraction capabilities. However, these methods face challenges in dealing with the complex inter- and intra-class dependencies inherent in different arrhythmias. We propose a classification model for electrocardiograms based on the Graph Convolutional Neural Network (EGCNet) to address this. In our EGCNet framework, we first employ the row and column 1-dimensional residual convolution to learn intra-class dependency. Then, we construct a hierarchical graph structure to model the inter-class dependency. In such a graph, each node represents one type of disease, and the edges describe the relationship between diseases. To reduce the complexity of inter-class dependency, we build a hierarchical graph structure by treating similar nodes as the same type. Thus, we can learn the discriminant dependency in the higher-level graph space and feed this high-level knowledge back to the lower-level graph space. Finally, we evaluated our method using publicly available datasets from the PhysioNet/Computing Challenge 2020 ECG data. For instance, our model achieved robust performance in the 24-label experiment with a precision of 0.753 and an F1 score of 0.722.

Compact feature representation of person image is important for person re-identification (Re-ID) task. Recently, part-based representation models have been widely studied for extracting the more compact and robust feature representation for person image to improve person Re-ID results. However, existing part-based representation models mostly extract the features of different parts independently which ignore the spatial relationship information among different parts. To address this issue, in this paper we propose a novel deep learning framework, named Part-based Hierarchical Graph Convolutional Network (PH-GCN) for person Re-ID problem. Given a person image, PH-GCN first constructs a hierarchical graph to represent the spatial relationships among different parts. Then, both local and global feature learning is achieved by the feature information passing in PH-GCN, which takes the information of other parts into account for part feature representation. Finally, a perceptron layer is adopted for the final person part label prediction and re-identification. The proposed framework provides a general solution that integrates <i>local</i>, <i>global</i> and <i>structural</i> feature learning simultaneously in a unified end-to-end network representation and learning. Extensive experiments on several widely used benchmark datasets demonstrate the effectiveness and benefits of the proposed PH-GCN approach for person Re-ID task.

Numerous patch-based methods have recently been proposed for histological image based breast cancer classification. However, their performance could be highly affected by ignoring spatial contextual information in the whole slide image (WSI). To address this issue, we propose a novel hierarchical Graph V-Net by integrating 1) patch-level pre-training and 2) context-based fine-tuning, with a hierarchical graph network. Specifically, a semi-supervised framework based on knowledge distillation is first developed to pre-train a patch encoder for extracting disease-relevant features. Then, a hierarchical Graph V-Net is designed to construct a hierarchical graph representation from neighboring/similar individual patches for coarse-to-fine classification, where each graph node (corresponding to one patch) is attached with extracted disease-relevant features and its target label during training is the average label of all pixels in the corresponding patch. To evaluate the performance of our proposed hierarchical Graph V-Net, we collect a large WSI dataset of 560 WSIs, with 30 labeled WSIs from the BACH dataset (through our further refinement), 30 labeled WSIs and 500 unlabeled WSIs from Yunnan Cancer Hospital. Those 500 unlabeled WSIs are employed for patch-level pre-training to improve feature representation, while 60 labeled WSIs are used to train and test our proposed hierarchical Graph V-Net. Both comparative assessment and ablation studies demonstrate the superiority of our proposed hierarchical Graph V-Net over state-of-the-art methods in classifying breast cancer from WSIs. The source code and our annotations for the BACH dataset have been released at https://github.com/lyhkevin/Graph-V-Net.

In this paper we present a new approach towards the extraction of affine image regions based on detecting shape-stable boundaries from a multi-scale image representation. We construct an affine morphological scale space (AMSS) representation [1], which performs anisotropic diffusion while preserving boundaries and being invariant to affine transformations. We extract the transition boundaries of the diffusivity velocity map and track their evolution at each level of the scale-space. We then determine the stability of the boundary shape through a minimization process over different scales. Unlike most state of the art detectors which use the Gaussian scale space for multi-scale image representation, our approach is intrinsically affine invariant. We evaluate our detector by measuring repeatability of regions in transformed images of the same scene and comparing it to the state-of-the-art region detectors [2].

Modeling the evolution process of industrial cascading faults by capturing causal relationships has achieved widespread attention in the attack detection and security control field. However, the highly coupled nature of numerous sensors and control loops in complex systems makes it difficult for existing root cause identification (RCI) methods to accurately and clearly characterize the fault-induced causal relationships. To address this challenge, this article proposes the hierarchical causal graph neural networks (HCGNNs) to reveal the root cause in an end-to-end manner with the aid of cascading failure analysis. First, the cause–effect weighted adjacency matrix is designed to expand the connections between nodes from correlations to causalities in a graph neural network (GNN). On this basis, the proposed hierarchical network structure can collaboratively learn the intraunit and interunit causalities, which significantly reduces the spurious and redundant causal relations while improving the modeling efficiency. Additionally, a new cascading failure analysis method is formulated to quantify the root cause and clarify propagation paths from a more objective perspective. Finally, the effectiveness and superiority of the proposed method are verified by the Tennessee Eastman platform and the real-world coal mill group.

In recent years, graph neural network is used to process graph data and has been successfully applied to graph node classification task. Due to the complexity of graph structure and the difficulty of obtaining node labels, node classification in datasets with fewer labels becomes a challenge. Existing node classification problems with few-label nodes datasets usually use neighbor aggregation schemes. However, these methods lack the “graph pooling mechanism,” which makes these models impossible to make full use of the global graph information. To solve the problem, we propose a Hierarchical Structure-Feature aware Graph Neural Network (HSFGNN) model. The model learns node features by a hierarchical mechanism that repeatedly utilizes coarsening and refining methods at different levels. Firstly, we use topological information, node characteristics and connectivity of the graph to construct an intuitive and effective coarsening part based on multivariable analysis technology. Secondly, we aggregate the unselected nodes’ features into the top-k nodes, so that the coarser nodes contain more abundant valid graph information. Finally, we propose an additional attention mechanism to obtain more accurate final representations of nodes. Experimental results show that the HSFGNN model achieves outstanding classification accuracy on the dataset with less labeled data.