Optimizing Data Distribution and Kernel Performance for Efficient Training of Chemistry Foundation Models: A Case Study with MACE

Traditional Graph Neural Network (GNN) learning patterns can only achieve optimal performance under the assumption of independent and identically distributed data. However, in real-world scenarios, numerous unpredictable factors give rise to the Out-of-Distribution (OOD) problem. Moreover, this distribution discrepancy may lead to unreliable predictions by GNNs models within unknown OOD domains. Therefore, this paper proposes a machine learning method to enhance the stability of GNN cross-site classification from the perspective of stable learning. The aim is to extract genuine causal features while eliminating spurious causal features. By introducing a feature sample weighting decorrelation technique in the random Fourier transform space and combining it with a baseline GNN model, a Stable-GNN model and a constrained sampling weight gradient update algorithm are designed. Its theoretical proof indicates that this algorithm can ensure the decrease in the loss, thus remedying the shortcomings of the original algorithm in weight updating. While guaranteeing predictive performance on the training distribution data, it also reduces prediction bias on data from unseen test distributions. Compared with the GNN model and the Stable-GNN model (S-GNN), the experimental results of the S-GNN model demonstrate that it not only surpasses the current state-of-the-art GNN models, but also offers a flexible framework for strengthening existing GNNs.

Graph neural networks (GNNs) have been widely used to learn node representations from graph data in an unsupervised way for downstream tasks. However, when applied to detect anomalies (e.g., outliers, unexpected density), they deliver unsatisfactory performance as existing loss functions fail. For example, any loss based on random walk (RW) algorithms would no longer work because the assumption that anomalous nodes were close with each other could not hold. Moreover, the nature of class imbalance in anomaly detection tasks brings great challenges to reduce the prediction error. In this work, we propose a novel loss function to train GNNs for anomaly-detectable node representations. It evaluates node similarity using global grouping patterns discovered from graph mining algorithms. It can automatically adjust margins for minority classes based on data distribution. Theoretically, we prove that the prediction error is bounded given the proposed loss function. We empirically investigate the GNN effectiveness of different loss variants based on different algorithms. Experiments on two real-world datasets show that they perform significantly better than RW-based loss for graph anomaly detection.

Graph neural networks (GNNs) emerge as a powerful approach to process non-euclidean data structures and have been proved powerful in various application domains such as social networks and e-commerce. While such graph data maintained in real-world systems can be extremely large and sparse, thus employing GNNs to deal with them requires substantial computational and memory overhead, which induces considerable energy and resource cost on CPUs and GPUs. In this article, we present a specialized accelerator architecture, EnGN, to enable high-throughput and energy-efficient processing of large-scale GNNs. The proposed EnGN is designed to accelerate the three key stages of GNN propagation, which is abstracted as common computing patterns shared by typical GNNs. To support the key stages simultaneously, we propose the ring-edge-reduce(RER) dataflow that tames the poor locality of sparsely-and-randomly connected vertices, and the RER PE-array to practice RER dataflow. In addition, we utilize a graph tiling strategy to fit large graphs into EnGN and make good use of the hierarchical on-chip buffers through adaptive computation reordering and tile scheduling. Overall, EnGN achieves performance speedup by 1802.9X, 19.75X, and 2.97X and energy efficiency by 1326.35X, 304.43X, and 6.2X on average compared to CPU, GPU, and a state-of-the-art GCN accelerator HyGCN, respectively.

The development of accurate and explicable machine learning models to predict the properties of topologically complex systems is a challenge in materials science. Porous organic cages, a class of polycyclic molecular materials, have potential application in molecular separations, catalysis and encapsulation. For most applications of porous organic cages, having a permanent internal cavity in the absence of solvent, a property termed “shape persistence” is critical. Here, we report the development of Graph Neural Networks (GNNs) to predict the shape persistence of organic cages. Graph neural networks are a class of neural networks where the data, in our case that of organic cages, are represented by graphs. The performance of the GNN models was measured against a previously reported computational database of organic cages formed through a range of [4 + 6] reactions with a variety of reaction chemistries. The reported GNNs have an improved prediction accuracy and transferability compared to random forest predictions. Apart from the improvement in predictive power, we explored the explicability of the GNNs by computing the integrated gradient of the GNN input. The contribution of monomers and molecular fragments to the shape persistence of the organic cages could be quantitatively evaluated with integrated gradients. With the added explicability of the GNNs, it was possible not only to accurately predict the property of organic materials, but also to interpret the predictions of the deep learning models and provide structural insights for the discovery of future materials.

SGNN-T: Space graph neural network coupled transformer for molecular property prediction

Graph Neural Networks (GNNs) have achieved promising results in numerous applications. To ensure accurate predictions, GNNs may exploit user data beyond the scope of consent, raising significant concerns regarding data usage and privacy. A critical task, therefore, is data usage privacy auditing, which aims to infer whether a data subject's information was improperly used during model training. While Membership Inference Attacks (MIA) are commonly used to audit data usage in fully supervised settings, most GNNs operate under a semi-supervised paradigm, where nodes fall into three distinct categories with varying degrees of involvement in training. This heterogeneity introduces a key challenge for applying traditional binary MIA methods. To address this, we study ternary data usage privacy auditing for semi-supervised GNNs in a black-box setting using a shadow training-based MIA framework. We identify that the main obstacle lies in the effects of message passing, which entangles node representations across categories. To tackle this, we propose MPMIA--a novel Message Passing-aware Membership Inference Attack--which leverages point-wise KL divergence to quantify and exploit the influence of message passing on node embeddings. Extensive experiments validate the effectiveness of MPMIA, showing clear improvements over baseline methods in accurately auditing data usage within semi-supervised GNNs.

Concept drift is the phenomenon where the data distribution in a data stream changes over time. It is a ubiquitous problem in the real-world, for example, a traffic accident would cause a jam in a certain period, leading to a distribution change in traffic speed. Most research in the concept drift field focuses on single data stream, however, few of them consider multi-stream environments which are more in line with the application needs. To fill this gap, we propose a multi-stream prediction setting and a multi-stream concept drift self-adaptation framework using graph neural network, named SAGN. In SAGN, we reconsider the learning procedure of GNN-based predictors from an aspect of concept drift adaptation for multi-stream. By this design, the prediction task is converted into online streaming data tasks in sub-graphs. Each sub-graph corresponds to an adaptation target and will be updated over time. In this way, locally we can overcome drift in each sub-graph by a designed adaptation technique, and globally the correlation between different data streams is well-preserved as a graph structure. Therefore, whether drift occurs or not, in one or several streams, SAGN can provide consistently accurate prediction results. We comprehensively tested SAGN on both synthetic and real-world, drift and non-drift data in the multi-step prediction task. The experiment results show that SAGN is able to achieve state-of-the-art performance in most cases.

Node representation learning on attributed graphs-whose nodes are associated with rich attributes (e.g., texts and protein sequences)-plays a crucial role in many important downstream tasks. To encode the attributes and graph structures simultaneously, recent studies integrate pre-trained models with graph neural networks (GNNs), where pre-trained models serve as node encoders (NEs) to encode the attributes. As jointly training large NEs and GNNs on large-scale graphs suffers from severe scalability issues, many methods propose to train NEs and GNNs separately. Consequently, they do not take feature convolutions in GNNs into consideration in the training phase of NEs, leading to a significant learning bias relative to the joint training. To address this challenge, we propose an efficient label regularization technique, namely Label Deconvolution (LD), to alleviate the learning bias by a novel and highly scalable approximation to the inverse mapping of GNNs. The inverse mapping leads to an objective function that is equivalent to that by the joint training, while it can effectively incorporate GNNs in the training phase of NEs against the learning bias. More importantly, we show that LD converges to the optimal objective function values by the joint training under mild assumptions. Experiments demonstrate LD significantly outperforms state-of-the-art methods on Open Graph Benchmark datasets.

We introduce , an open-source Python library designed for mesh agglomeration in both two- and three-dimensions, based on employing Graph Neural Networks (GNN). serves as a comprehensive solution for training a variety of GNN models, integrating deep learning and other advanced algorithms such as METIS and k-means to facilitate mesh agglomeration and quality metric computation. The library’s introduction is outlined through its code structure and primary features. The GNN framework adopts a graph bisection methodology that capitalizes on connectivity and geometric mesh information via SAGE convolutional layers, in line with the methodology proposed in (Antonietti and Manuzzi in J Comput Phys 452:110900, 2022; Antonietti et al. in Polytopal mesh agglomeration via geometrical deep learning for three-dimensional heterogeneous domains, arXiv:2406.10587 , 2024). Additionally, the proposed library incorporates reinforcement learning to enhance the accuracy and robustness of the model initially suggested in [1, 2] for predicting coarse partitions within a multilevel framework. A detailed tutorial is provided to guide the user through the process of mesh agglomeration and the training of a GNN bisection model. We present several examples of mesh agglomeration conducted by , demonstrating the library’s applicability across various scenarios. Furthermore, the performance of the newly introduced models is contrasted with that of METIS and k-means, illustrating that the proposed GNN models are competitive regarding partition quality and computational efficiency. Finally, we exhibit the versatility of ’s interface through its integration with , an open-source library implementing discontinuous Galerkin methods on polytopal grids for the numerical discretization of multiphysics differential problems.

MIGP: Metapath Integrated Graph Prompt Neural Network

Federated Graph Learning (FGL) aims to learn graph learning models over graph data distributed in multiple data owners, which has been applied in various applications such as social recommendation and financial fraud detection. Inherited from generic Federated Learning (FL), FGL similarly has the data heterogeneity issue where the label distribution may vary significantly for distributed graph data across clients. For instance, a client can have the majority of nodes from a class, while another client may have only a few nodes from the same class. This issue results in divergent local objectives and impairs FGL convergence for node-level tasks, especially for node classification. Moreover, FGL also encounters a unique challenge for the node classification task: the nodes from a minority class in a client are more likely to have biased neighboring information, which prevents FGL from learning expressive node embeddings with Graph Neural Networks (GNNs). To grapple with the challenge, we propose FedSpray, a novel FGL framework that learns local class-wise structure proxies in the latent space and aligns them to obtain global structure proxies in the server. Our goal is to obtain the aligned structure proxies that can serve as reliable, unbiased neighboring information for node classification. To achieve this, FedSpray trains a global feature-structure encoder and generates unbiased soft targets with structure proxies to regularize local training of GNN models in a personalized way. We conduct extensive experiments over four datasets, and experiment results validate the superiority of FedSpray compared with other baselines. Our code is available at https://github.com/xbfu/FedSpray.

Graph Neural Networks (GNN) are a promising technique for bridging differential programming and combinatorial domains. GNNs employ trainable modules which can be assembled in different configurations that reflect the relational structure of each problem instance. In this paper, we show that GNNs can learn to solve, with very little supervision, the decision variant of the Traveling Salesperson Problem (TSP), a highly relevant NP-Complete problem. Our model is trained to function as an effective message-passing algorithm in which edges (embedded with their weights) communicate with vertices for a number of iterations after which the model is asked to decide whether a route with cost < C exists. We show that such a network can be trained with sets of dual examples: given the optimal tour cost C∗, we produce one decision instance with target cost x% smaller and one with target cost x% larger than C∗. We were able to obtain 80% accuracy training with −2%,+2% deviations, and the same trained model can generalize for more relaxed deviations with increasing performance. We also show that the model is capable of generalizing for larger problem sizes. Finally, we provide a method for predicting the optimal route cost within 2% deviation from the ground truth. In summary, our work shows that Graph Neural Networks are powerful enough to solve NP-Complete problems which combine symbolic and numeric data.

Accelerated prediction of molecular properties for per- and polyfluoroalkyl substances using graph neural networks with adjacency-free message passing.

This paper focuses on the dissemination of city brand visual symbols in a multi-platform social media environment. Addressing the shortcomings of existing research in maintaining semantic consistency and cross-platform dissemination efficiency, we propose a semantic dissemination path modeling and optimization method based on graph neural networks. This study first constructs a semantic graph for symbolic dissemination, providing a structured representation of users, groups, and visual symbol nodes. A framework combining graph convolutional networks and graph attention networks is introduced to capture high-order dependencies within complex networks. The experimental design encompasses multi-platform data collection, feature extraction, and model training. Results demonstrate that this method significantly outperforms baseline models in terms of dissemination coverage, semantic consistency index, and cross-platform dissemination efficiency. Specifically, the model improves coverage by over 15%, raises the semantic consistency index to 0.83, and reduces the time required for cross-platform dissemination by approximately 22%. These results fully demonstrate the effectiveness and applicability of the proposed method in enhancing the international dissemination of city brands. This paper's innovation lies in combining semiotic analysis with graph neural network modeling to propose a quantitative and optimizable dissemination path framework. This not only expands the research horizons of computational communication but also provides new methodological support and practical references for the internationalization of brands in cities such as the Guangdong-Hong Kong-Macao Greater Bay Area.

Enhancing the properties of advanced drug candidates is aided by the direct incorporation of specific chemical groups, avoiding the need to construct the entire compound from the ground up. Nevertheless, their chemical intricacy often poses challenges in predicting reactivity for C-H activation reactions and planning their synthesis. We adopted a reaction screening approach that combines high-throughput experimentation (HTE) at a nanomolar scale with computational graph neural networks (GNNs). This approach aims to identify suitable substrates for late-stage C-H alkylation using Minisci-type chemistry. GNNs were trained using experimentally generated reactions derived from in-house HTE and literature data. These trained models were then used to predict, in a forward-looking manner, the coupling of 3180 advanced heterocyclic building blocks with a diverse set of sp3-rich carboxylic acids. This predictive approach aimed to explore the substrate landscape for Minisci-type alkylations. Promising candidates were chosen, their production was scaled up, and they were subsequently isolated and characterized. This process led to the creation of 30 novel, functionally modified molecules that hold potential for further refinement. These results positively advocate the application of HTE-based machine learning to virtual reaction screening.