Node Context Selection in Transformer-Based Graph Representation Learning Models

We investigate works under the propagation-based fake news detection domain, which recently seeks to improve performance through the use of Graph Neural Networks (GNNs). Generally, existing works argue that using GNNs can give results superior to what was obtained using classic graph-based methods. We agree with this argument given that GNNs are capable of gaining superior performance by leveraging node features. But we argue that existing works haven't identified the fact that the expressivity of GNNs is limited and bounded by node features. Existing works do not acknowledge that, by utilizing GNNs, they implicitly assume node features are strongly correlated to node labels. There are evidence that node features that have been employed do not necessarily correlate to node label. Instead of having a profound theoretical motivation, they have empirically observed that focusing on nodes features with strong feature-label correlation can increase predictive capability. This is a sub-optimal approach to view this problem, in fact, we argue that finding node features based on correlation is not practical or effective. Our first contribution is shifting readers from a node-level view i.e correlating node features with labels, to a graph-level view. In the graph-level view, we exploit the relationship between graph isomorphism and GNNs' expressivity which can be utilized to well understand and interpret the relation between node features and GNNs' expressivity. We conduct a wide range of experiments on basis of both node-level view and graph-level view and found graph-level view is more interpretable and strongly matches with results. Further, we gained insights on node features that wouldn't be obtainable by a node-level view. In order to have a fair and comprehensive analysis of node features, we built a unified dataset that includes a wide range of node features. Our results indicate, as we improve model accuracy on basis of the graph level view, models' generalizability decreases. We provide our hypothesis for this performance trade-off on the basis of the graph-level view. Our results and insights call for a much broader discussion on whether any sort of filtering method is effective. So, we conclude our work by providing readers with possible solutions that can be helpful to find harmony between node features and GNNs' expressivity.

Graph neural networks have been paid a lot attentions in recent years since many real-word data can naturally be represented by graph structures. Graph neural networks such as GCNs and GATs mainly focus on node features while ignoring edge features in graphs. However, in many graph structure data such as knowledge graphs, social networks, edge features are also important as they contain vital information about relations between nodes which are commonly ignored or simplified into binary or scalar values by existing methods. In this work we build a novel learning method on graphs called GNN-EE, i.e. GNN with Edge Enhanced, which takes both the node features and edge features into account while updating representations of graph components and can be applied to most of the common graph neural networks such as GCNs and GATs. Our GNN-EE method fits in the message-passing framework and thus is easy to generalize. In addition, we extend the random-walk-based algorithms on graphs so that they can consider both node and edge features on graphs. We use those random-walk-based algorithms as a pre-training method on graph with few initial features. We demonstrate the effectiveness and flexibility of our GNN-EE method through entity classification tasks and graph classification tasks.

Peptides are crucial in biological processes and therapeutic applications. Given their importance, advancing our ability to predict peptide properties is essential. In this study, we introduce Multi-Peptide, an innovative approach that combines transformer-based language models with graph neural networks (GNNs) to predict peptide properties. We integrate PeptideBERT, a transformer model specifically designed for peptide property prediction, with a GNN encoder to capture both sequence-based and structural features. By employing a contrastive loss framework, Multi-Peptide aligns embeddings from both modalities into a shared latent space, thereby enhancing the transformer model's predictive accuracy. Evaluations on hemolysis and nonfouling data sets demonstrate Multi-Peptide's robustness, achieving state-of-the-art 88.057% accuracy in hemolysis prediction. This study highlights the potential of multimodal learning in bioinformatics, paving the way for accurate and reliable predictions in peptide-based research and applications.

Graph neural networks (GNNs) with missing node features have recently received increasing interest. Such missing node features seriously hurt the performance of the existing GNNs. Some recent methods have been proposed to reconstruct the missing node features by the information propagation among nodes with known and unknown attributes. Although these methods have achieved superior performance, how to exactly exploit the complex data correlations among nodes to reconstruct missing node features is still a great challenge. To solve the above problem, we propose a self-supervised guided hypergraph feature propagation (SGHFP). Specifically, the feature hypergraph is first generated according to the node features with missing information. And then, the reconstructed node features produced by the previous iteration are fed to a two-layer GNNs to construct a pseudo-label hypergraph. Before each iteration, the constructed feature hypergraph and pseudo-label hypergraph are fused effectively, which can better preserve the higher-order data correlations among nodes. After then, we apply the fused hypergraph to the feature propagation for reconstructing missing features. Finally, the reconstructed node features by multi-iteration optimization are applied to the downstream semi-supervised classification task. Extensive experiments demonstrate that the proposed SGHFP outperforms the existing semi-supervised classification with missing node feature methods.

Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. The key to the GNNs is adopting the neural message-passing paradigm with two stages: aggregation and update. The current design of GNNs considers the topology information in the aggregation stage. However, in the updating stage, all nodes share the same updating function. The identical updating function treats each node embedding as i.i.d. random variables and thus ignores the implicit relationships between neighborhoods, which limits the capacity of the GNNs. The updating function is usually implemented with a linear transformation followed by a non-linear activation function. To make the updating function topology-aware, we inject the topological information into the non-linear activation function and propose Graph-adaptive Rectified Linear Unit (GReLU), which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. The parameters of GReLU are obtained from a hyperfunction based on both node features and the corresponding adjacent matrix. To reduce the risk of overfitting and the computational cost, we decompose the hyperfunction as two independent components for nodes and features respectively. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.

SMGCL: Semi-supervised Multi-view Graph Contrastive Learning

A spatial–temporal graph neural network framework for automated software bug triaging

We propose an end-to-end framework based on a Graph Neural Network (GNN) to balance the power flows in energy grids. The balancing is framed as a supervised vertex regression task, where the GNN is trained to predict the current and power injections at each grid branch that yield a power flow balance. By representing the power grid as a line graph with branches as vertices, we can train a GNN that is accurate and robust to changes in topology. In addition, by using specialized GNN layers, we are able to build a very deep architecture that accounts for large neighborhoods on the graph, while implementing only localized operations. We perform three different experiments to evaluate: i) the benefits of using localized rather than global operations and the tendency of deep GNN models to oversmooth the quantities on the nodes; ii) the resilience to perturbations in the graph topology; and iii) the capability to train the model simultaneously on multiple grid topologies and the consequential improvement in generalization to new, unseen grids. The proposed framework is efficient and, compared to other solvers based on deep learning, is robust to perturbations not only to the physical quantities on the grid components, but also to the topology.

A Novel Privacy-Preserving Graph Convolutional Network via Secure Matrix Multiplication

Graph neural networks(GNNs) have been developed to solve challenging resource allocation (RA) problems, which leads to hopeful results in small and simple communication networks. Due to the inevitable heterogeneity of modern networks, it motivated researchers to develop the heterogeneous graph neural networks (HetGNNs) model for the RA problem of heterogeneous networks. However, node features and edge features are usually ignored by the most extant deep models, limited the performance when the size of the hidden layer in the network is larger than that of the node and edge features. In this paper, the power control or beamforming (PC/BF) in heterogeneous device-to-device (D2D) networks is focused, and proposed a HetGNN for the issue. Extensive simulations show that the proposed approach, matching or even outperforming state-of-the-art learning-based benchmarks.

Recent advances in Graph Neural Networks (GNNs) have changed the landscape of modern graph analytics. The complexity of GNN training and the scalability challenges have also sparked interest from the systems community, with efforts to build systems that provide higher efficiency and schemes to reduce costs. However, we observe that many such systems basically "reinvent the wheel" of much work done in the database world on scalable graph analytics engines. Further, they often tightly couple the scalability treatments of graph data processing with that of GNN training, resulting in entangled complex problems and systems that often do not scale well on one of those axes. In this paper, we ask a fundamental question: How far can we push existing systems for scalable graph analytics and deep learning (DL) instead of building custom GNN systems? Are compromises inevitable on scalability and/or runtimes? We propose Lotan, the first scalable and optimized data system for full-batch GNN training with decoupled scaling that bridges the hitherto siloed worlds of graph analytics systems and DL systems. Lotan offers a series of technical innovations, including re-imagining GNN training as query plan-like dataflows, execution plan rewriting, optimized data movement between systems, a GNN-centric graph partitioning scheme, and the first known GNN model batching scheme. We prototyped Lotan on top of GraphX and PyTorch. An empirical evaluation using several real-world benchmark GNN workloads reveals a promising nuanced picture: Lotan significantly surpasses the scalability of state-of-the-art custom GNN systems, while often matching or being only slightly behind on time-to-accuracy metrics in some cases. We also show the impact of our system optimizations. Overall, our work shows that the GNN world can indeed benefit from building on top of scalable graph analytics engines. Lotan's new level of scalability can also empower new ML-oriented research on ever-larger graphs and GNNs.

In recent years, the transformer model has been widely used in computer-vision tasks and has achieved impressive results. Unfortunately, these transformer-based models have the common drawback of having many parameters and a large memory footprint, causing them to be difficult to deploy on mobiles as lightweight convolutional neural networks. To address these issues, a Vision Transformer (ViT) model, named the lightweight Vision Transformer (LW-ViT) model, is proposed to reduce the complexity of the transformer-based model. The model is applied to offline handwritten Chinese character recognition. The design of the LW-ViT model is inspired by MobileViT. The lightweight ViT model reduces the number of parameters and FLOPs by reducing the number of transformer blocks and the MV2 layer based on the overall framework of the MobileViT model. The number of parameters and FLOPs for the LW-ViT model was 0.48 million and 0.22 G, respectively, and it ultimately achieved a high recognition accuracy of 95.8% on the dataset. Furthermore, compared to the MobileViT model, the number of parameters was reduced by 53.8%, and the FLOPs were reduced by 18.5%. The experimental results show that the LW-ViT model has a low number of parameters, proving the correctness and feasibility of the proposed model.

Graph Neural Networks (GNNs) are the current state-of-the-art models in learning node representations for many predictive tasks on graphs. Typically, GNNs reuses the same set of model parameters across all nodes in the graph to improve the training efficiency and exploit the translationally-invariant properties in many datasets. However, the parameter sharing scheme prevents GNNs from distinguishing two nodes that are isomorphic and that the translation invariance property may not exhibit in real-world graphs. In this paper, we present Graph Representation Learning with Adaptive Mixtures (GRAM), a novel approach for learning node representations in a graph by introducing multiple independent GNN models and a trainable mixture distribution for each node. GRAM is a general framework that can be readily embedded into existing models for enhancing representation learning. It allows for adaptive node clustering with mixtures and joint optimization of all GNN models simultaneously. To achieve this, we develop an Expectation-Maximization algorithm with stochastic gradient descent in a scalable manner. Specifically, in the E-Step, we approximate the posterior probability distribution of the latent cluster membership based on the k-th independent GNN models in the last iteration. In the M-step, we update all GNN models and the mixture parameters based on the refined latent cluster membership for downstream tasks, e.g., node classification. To fully exploit the graph structural information, we further implement a regularization term on unlabeled nodes, which encourages adjacent nodes to have the same mixture distribution. By jointly optimizing the model in an end-to-end manner, GRAM learns to capture different abstract patterns on graphs for different node clusters, which yields discriminative representation for each node. We evaluate GRAM on five benchmark datasets with extensive experiments. GRAM is demonstrated to consistently boost state-of-the-art GNN variants in node classification tasks.

A Comprehensive Survey on Deep Graph Representation Learning

Graphs help to define the relationships between entities in the data. These relationships, represented by edges, often provide additional context information which can be utilised to discover patterns in the data. Graph Neural Networks (GNNs) employ the inductive bias of the graph structure to learn and predict on various tasks. The primary operation of graph neural networks is the feature aggregation step performed over neighbours of the node based on the structure of the graph. In addition to its own features, for each hop, the node gets additional combined features from its neighbours. These aggregated features help define the similarity or dissimilarity of the nodes with respect to the labels and are useful for tasks like node classification. However, in real‐world data, features of neighbours at different hops may not correlate with the node's features. Thus, any indiscriminate feature aggregation by GNN might cause the addition of noisy features leading to degradation in model's performance. In this work, we show that selective aggregation of node features from various hops leads to better performance than default aggregation on the node classification task. Furthermore, we propose a Dual‐Net GNN architecture with a classifier model and a selector model. The classifier model trains over a subset of input node features to predict node labels while the selector model learns to provide optimal input subset to the classifier for the best performance. These two models are trained jointly to learn the best subset of features that give higher accuracy in node label predictions. With extensive experiments, we show that our proposed model outperforms both feature selection methods and state‐of‐the‐art GNN models with remarkable improvements up to 27.8%.