Task-Aware Retrieval Augmentation for Dynamic Recommendation

TodyNet: Temporal dynamic graph neural network for multivariate time series classification

Graph Neural Networks (GNNs) have been widely used for modeling graph-structured data. With the development of numerous GNN variants, recent years have witnessed groundbreaking results in improving the scalability of GNNs to work on static graphs with millions of nodes. However, how to instantly represent continuous changes of large-scale dynamic graphs with GNNs is still an open problem. Existing dynamic GNNs focus on modeling the periodic evolution of graphs, often on a snapshot basis. Such methods suffer from two drawbacks: first, there is a substantial delay for the changes in the graph to be reflected in the graph representations, resulting in losses on the model's accuracy; second, repeatedly calculating the representation matrix on the entire graph in each snapshot is predominantly time-consuming and severely limits the scalability. In this paper, we propose Instant Graph Neural Network (InstantGNN), an incremental computation approach for the graph representation matrix of dynamic graphs. Set to work with dynamic graphs with the edge-arrival model, our method avoids time-consuming, repetitive computations and allows instant updates on the representation and instant predictions. Graphs with dynamic structures and dynamic attributes are both supported. The upper bounds of time complexity of those updates are also provided. Furthermore, our method provides an adaptive training strategy, which guides the model to retrain at moments when it can make the greatest performance gains. We conduct extensive experiments on several real-world and synthetic datasets. Empirical results demonstrate that our model achieves state-of-the-art accuracy while having orders-of-magnitude higher efficiency than existing methods.

Graph Neural Networks (GNNs) have become the de facto standard for representation learning on topological graphs, which usually derive effective node representations via message passing from neighborhoods. Although GNNs have achieved great success, previous models are mostly confined to static and homogeneous graphs. However, there are multiple dynamic interactions between different-typed nodes in real-world scenarios like academic networks and e-commerce platforms, forming temporal heterogeneous graphs (THGs). Limited work has been done for representation learning on THGs and the challenges are in two aspects. First, there are abundant dynamic semantics between nodes while traditional techniques like meta-paths can only capture static relevance. Second, different semantics on THGs have often mutually evolved with each other over time, making it more difficult than dynamic homogeneous graph modeling. To address this problem, here we propose the Dynamic Meta-path guided temporal heterogeneous Graph Neural Networks (DyMGNNs). To handle the dynamic semantics, we introduce the concept of dynamic meta-path which is a common base for temporal semantic search engines, and then adopt the temporal importance sampling to extract neighborhoods with temporal bias. Focusing on mutual evolution, we design the heterogeneous mutual evolution attention mechanism, which can model the fine-grained interplay of semanticlevel preferences for each node. Extensive experiments on three real-world datasets for node classification and temporal link prediction demonstrate that our method consistently outperforms state-of-the-art alternatives.

Graph Neural Networks (GNNs) have emerged in recent years as a powerful tool to learn tasks across a wide range of graph domains in a data-driven fashion. Based on a message passing mechanism, GNNs have gained increasing popularity due to their intuitive formulation, closely linked to the Weisfeiler–Lehman (WL) test for graph isomorphism, to which they were demonstrated to be equivalent (Morris et al., 2019 and Xu et al., 2019). From a theoretical point of view, GNNs have been shown to be universal approximators, and their generalization capability — related to the Vapnik Chervonekis (VC) dimension (Scarselli et al., 2018) — has recently been investigated for GNNs with piecewise polynomial activation functions (Morris et al., 2023). The aim of our work is to extend this analysis on the VC dimension of GNNs to other commonly used activation functions, such as the sigmoid and hyperbolic tangent, using the framework of Pfaffian function theory. Bounds are provided with respect to the architecture parameters (depth, number of neurons, input size) as well as with respect to the number of colors resulting from the 1–WL test applied on the graph domain. The theoretical analysis is supported by a preliminary experimental study.

Graphs are essential representations of many real-world data such as social networks. Recent years have witnessed the increasing efforts made to extend the neural network models to graph-structured data. These methods, which are usually known as the graph neural networks, have been applied to advance many graphs related tasks such as reasoning dynamics of the physical system, graph classification, and node classification. Most of the existing graph neural network models have been designed for static graphs, while many real-world graphs are inherently dynamic. For example, social networks are naturally evolving as new users joining and new relations being created. Current graph neural network models cannot utilize the dynamic information in dynamic graphs. However, the dynamic information has been proven to enhance the performance of many graph analytic tasks such as community detection and link prediction. Hence, it is necessary to design dedicated graph neural networks for dynamic graphs. In this paper, we propose DGNN, a new {\bf D}ynamic {\bf G}raph {\bf N}eural {\bf N}etwork model, which can model the dynamic information as the graph evolving. In particular, the proposed framework can keep updating node information by capturing the sequential information of edges (interactions), the time intervals between edges and information propagation coherently. Experimental results on various dynamic graphs demonstrate the effectiveness of the proposed framework.

This paper proposes a physically constrained dynamic attention graph neural network (PC-DAGNN). Its core innovations lie in: first, constructing a physical prior constraint module to transform experimental theoretical formulas into hard-constrained loss terms, ensuring that the output conforms to physical laws; second, designing a dynamic attention graph neural network module to capture the coupling relationship between error sources at different experimental stages in real time. To verify the algorithm's performance, 10,000 sets of samples (8,000 training sets and 2,000 test sets, categorized as low, medium, and high error strength) containing random and systematic errors were generated using MATLAB, using the photoelectric effect and free fall experiments as experimental scenarios. The results were compared with the least squares (LS) method, a BP neural network, and a generalized graph neural network (GNN). Results show that the PC-DAGNN achieves a maximum reduction of 66% in mean absolute error (MAE) compared to the comparison algorithms, a consistently high physical law compliance rate (PCR) above 96%, and a generalization error fluctuation (GEF) only 50%-60% of that of the comparison algorithms. Research has demonstrated that PC-DAGNN can achieve high-precision and highly stable error analysis while ensuring physical logic consistency, providing an effective technical solution for error correction in complex physics experiments.

Traffic forecasting plays an important role in intelligent transportation systems. However, the prediction task is highly challenging due to the mixture of global and local spatiotemporal dependencies involved in traffic data. Existing graph neural networks (GNNs) typically capture spatial dependencies with the predefined or learnable static graph structure, ignoring the hidden dynamic patterns in traffic networks. Meanwhile, most recurrent neural networks (RNNs) or convolutional neural networks (CNNs) cannot effectively capture temporal correlations, especially for long-term temporal dependencies. In this paper, we propose a spatial–temporal attention graph convolution network (STAGCN), which acquires a static graph and a dynamic graph from data without any prior knowledge. The static graph aims to model global space adaptability, and the dynamic graph is designed to capture local dynamics in the traffic network. A gated temporal attention module is further introduced for long-term temporal dependencies, where a causal-trend attention mechanism is proposed to increase the awareness of causality and local trends in time series. Extensive experiments on four real-world traffic flow datasets demonstrate that STAGCN achieves an outstanding prediction accuracy improvement over existing solutions.

Mutual funds are interconnected to each other through multiple types of links, including but not limited to co-investment, advisors, firms, and managers. These connections enable information flow among network entities, influence investment decisions, and ultimately impact mutual fund managers’ performance. In this paper, we propose a dynamic graph neural network approach to model these heterogeneous relationships and their contributions to mutual fund performance. Using the graph attention mechanism, our model learns latent embedding for mutual funds and their invested assets dynamically in each month and then uses the embedding to estimate future returns. Empirical analysis confirms that the proposed method outperforms the state-of-the-art DeepWalk model by 10%. Furthermore, this study also reveals the importance of networks in mutual fund performance. The inclusion of network connections in a feedforward machine learning model significantly increases the performance of the model by 118%. Finally, portfolio analysis and regression estimation on next month’s excess return show that the proposed approach has a significant economic contribution over current benchmark approaches.

Graph neural networks (GNNs) have achieved great success in many research areas ranging from traffic to computer vision. With increased interest in cloud-native applications, GNNs are increasingly being investigated to address various challenges in microservice architecture from prototype design to large-scale service deployment. To appreciate the big picture of this emerging trend, we provide a comprehensive review of recent studies leveraging GNNs for microservice-based applications. To begin, we identify the key areas in which GNNs are applied, and then we review in detail how GNNs can be designed to address the challenges in specific areas found in the literature. Finally, we outline potential research directions where GNN-based solutions can be further applied. Our research shows the popularity of leveraging convolutional graph neural networks (ConGNNs) for microservice-based applications in the current design of cloud systems and the emerging area of adopting spatio-temporal graph neural networks (STGNNs) and dynamic graph neural networks (DGNNs) for more advanced studies.

A spatial–temporal graph neural network framework for automated software bug triaging

Link prediction in Online Social Networks—OSNs—has been the focus of numerous studies in the machine learning community. A successful machine learning-based solution for this task needs to (i) leverage global and local properties of the graph structure surrounding links; (ii) leverage the content produced by OSN users; and (iii) allow their representations to change over time, as thousands of new links between users and new content like textual posts, comments, images and videos are created/uploaded every month. Current works have successfully leveraged the structural information but only a few have also taken into account the textual content and/or the dynamicity of network structure and node attributes. In this paper, we propose a methodology based on temporal graph neural networks to handle the challenges described above. To understand the impact of textual content on this task, we provide a novel pipeline to include textual information alongside the structural one with the usage of BERT language models, dense preprocessing layers, and an effective post-processing decoder. We conducted the evaluation on a novel dataset gathered from an emerging blockchain-based online social network, using a live-update setting that takes into account the evolving nature of data and models. The dataset serves as a useful testing ground for link prediction evaluation because it provides high-resolution temporal information on link creation and textual content, characteristics hard to find in current benchmark datasets. Our results show that temporal graph learning is a promising solution for dynamic link prediction with text. Indeed, combining textual features and dynamic Graph Neural Networks—GNNs—leads to the best performances over time. On average, the textual content can enhance the performance of a dynamic GNN by 3.1% and, as the collection of documents increases in size over time, help even models that do not consider the structural information of the network.

Graph neural networks (GNNs) have demonstrated their effectiveness in various tasks supported by their generalization capabilities. However, the current analysis of GNN generalization relies on the assumption that training and testing data are independent and identically distributed (i.i.d). This imposes limitations on the cases where a model mismatch exists when generating testing data. In this paper, we examine GNNs that operate on geometric graphs generated from manifold models, explicitly focusing on scenarios where there is a mismatch between manifold models generating training and testing data. Our analysis reveals the robustness of the GNN generalization in the presence of such model mismatch. This indicates that GNNs trained on graphs generated from a manifold can still generalize well to unseen nodes and graphs generated from a mismatched manifold. We attribute this mismatch to both node feature perturbations and edge perturbations within the generated graph. Our findings indicate that the generalization gap decreases as the number of nodes grows in the training graph while increasing with larger manifold dimension as well as larger mismatch. Importantly, we observe a trade-off between the generalization of GNNs and the capability to discriminate high-frequency components when facing a model mismatch. The most important practical consequence of this analysis is to shed light on the filter design of generalizable GNNs robust to model mismatch. We verify our theoretical findings with experiments on multiple real-world datasets.

Incorporating dynamic characteristics into Graph Neural Networks (GNNs) enhances the understanding of dynamic graph evolution, optimizing temporal-spatial representations for real-world dynamic network problems. However, existing discrete-time dynamic graphs (DTDGs) models face two crucial problems that are difficulty in capturing long-term dependencies and the unbalance of performance and computational efficiency. To address these issues, we propose the AGS-DGNN framework for modeling dynamic graphs which includes two components, one is a dynamic GNN model based on an adaptive gradient smoothing mechanism and meta-learning strategy to generate node embeddings and the other is a Transformer model for these node embedding sequences. Specifically, AGS-DGNN first computes the frame-wise loss for the current snapshot and applies an Exponential Moving Average (EMA) mechanism to smooth the loss gradients, thereby generating local EMA gradients. Then, the local EMA gradients from previous snapshots are aggregated to form a global EMA gradient, enabling effective gradient propagation along the temporal dimension under a meta-learning strategy. Finally, the model adaptively integrates the local and global EMA gradients using a dynamic adjustment factor to update the GNN. Additionally, a Transformer with multi-head self-attention mechanism is applied on the embedding sequence of snapshots to enhance the robustness and performance for the framework. Experiments on six public datasets show the advantage of our AGS-DGNN compared with existing baselines, where it has reached the optimum in sixteen out of eighteen performance metrics.

DyHDGE: Dynamic heterogeneous transaction graph embedding for safety-centric fraud detection in financial scenarios

With information explosion occurring in past decades, the rapid growth of papers published results in the rapid change of hot topics, especially in the biomedical domain. It turns out very hard for researchers who are interested in biomedical domain to track hot topics over time, as well as to predict the trends of them in the near future. Based on the above demand, it is important to have a model which is able to follow and predict the trend of hot topics continuously. Deep learning has been proven to be an efficient method to extract information from texts and use the information to predict the future trends. Under the thriving background of Deep Learning, Graph Neural Network (GNN) is able to capture the information from graph structures. There are various applications using GNN models, such as traffic flow prediction, chemical structure discovering, etc. In this research project, a dynamic spatio-temporal graph neural network is presented to keep track of the selected hot keywords and topics in the biomedical domain and predict the possible frequencies in the near future. The input of the model is obtained by extracting the monthly frequency information of selected keywords and topics from paper abstracts in PubMed, the largest biomedical literature collection. After training with data over a decade, the model is able to predict trends of selected hot keywords and topics in next 5 months. Thus, the presented model can help follow the trend of hot topics in the biomedical domain.