Molecular Representation Learning via Hierarchical Graph Transformer

Molecular property prediction with deep learning has accelerated drug discovery and retrosynthesis. However, the shortage of labeled molecular data and the challenge of generalizing across the vast chemical spaces pose significant hurdles for leveraging deep learning in molecular property prediction. This study proposes a self-supervised framework designed to acquire a Simplified Molecular Input Line Entry System (SMILES) representation, which we have dubbed Simple SMILES contrastive learning (SimSon). SimSon was pre-trained using unlabeled SMILES data through contrastive learning to grasp the SMILES representations. Our findings demonstrate that contrastive learning with randomized SMILES enriches the ability of the model to generalize and its robustness as it captures the global semantic context at the molecular level. In downstream tasks, SimSon performs competitively when compared to graph-based methods and even outperforms them on certain benchmark datasets. These results indicate that SimSon effectively captures structural information from SMILES, exhibiting remarkable generalization and robustness. The potential applications of SimSon extend to bioinformatics and cheminformatics, encompassing areas such as drug discovery and drug-drug interaction prediction. The source code is available at https://github.com/lee00206/SimSon.

Non-point source pollution from agricultural activities poses a significant threat to water quality by introducing excess nutrients like nitrogen into aquatic ecosystems, leading to issues such as eutrophication and groundwater contamination. In agricultural watersheds, nitrate transport involves intricate physical, chemical, and biological processes influenced by meteorological conditions, hydrological features, and spatial topologies, making accurate short-term predictions challenging. Traditional data-driven deep learning models often fail to incorporate physical constraints and complex spatiotemporal dynamics, limiting their interpretability and predictive accuracy. Here we show a hierarchical transformer and graph neural network model that accurately predicts watershed nitrate concentrations by integrating multi-source data and simulating pollutant migration. The model captures nonlinear multivariate temporal patterns through hierarchical transformers, fuses global meteorological and local hydrological features via neural networks, and models runoff topologies with physically constrained graph neural networks. For predicting the concentration changes of pollutants discharged from watersheds, it outperforms baselines like multi-layer perceptrons, recurrent neural networks, and long short-term memory networks, with state-of-the-art performance in root mean square error, mean absolute error, and R2. Ablation studies confirm the essential roles of multi-source data integration and watershed topological modeling in enhancing performance. This method of directly modeling physical processes by leveraging the characteristics of different neural network architectures opens up a new path for addressing the interpretability problem in neural earth system modeling, apart from the process-guided deep learning and differentiable modelling methods.

Molecular property prediction is an essential task in drug discovery. Most computational approaches with deep learning techniques either focus on designing novel molecular representation or combining with some advanced models together. However, researchers pay fewer attention to the potential benefits in massive unlabeled molecular data (e.g., ZINC). This task becomes increasingly challenging owing to the limitation of the scale of labeled data. Motivated by the recent advancements of pretrained models in natural language processing, the drug molecule can be naturally viewed as language to some extent. In this paper, we investigate how to develop the pretrained model BERT to extract useful molecular substructure information for molecular property prediction. We present a novel end‐to‐end deep learning framework, named Mol‐BERT, that combines an effective molecular representation with pretrained BERT model tailored for molecular property prediction. Specifically, a large‐scale prediction BERT model is pretrained to generate the embedding of molecular substructures, by using four million unlabeled drug SMILES (i.e., ZINC 15 and ChEMBL 27). Then, the pretrained BERT model can be fine‐tuned on various molecular property prediction tasks. To examine the performance of our proposed Mol‐BERT, we conduct several experiments on 4 widely used molecular datasets. In comparison to the traditional and state‐of‐the‐art baselines, the results illustrate that our proposed Mol‐BERT can outperform the current sequence‐based methods and achieve at least 2% improvement on ROC‐AUC score on Tox21, SIDER, and ClinTox dataset.

Machine Learning approaches are required to predict accurately on test samples that are distributionally different from training ones in the fields of drug discovery, computational biology, and cheminformatics. However, (i) labeled task-specific molecule data are often scarce, and (ii) poor generalization due to test molecules that are structurally different from those seen during training. To alleviate the problems, we propose a cloze-style self-supervised learning model (MolCloze) to obtain universal informative representations for molecular property prediction tasks. With carefully designed self-supervised tasks unifying generative- and discriminative-paradigm, MolCloze can learn rich structural and semantic information of molecules from enormous unlabelled molecular data. To capture such complex information, we design two novel strategies - Structural Fingerprint Tokenization (SFT) for better tokenizing molecule graphs, and Normalized Graph Raw Shortcut-connection (NGRS) for better latent representations by training a deeper model. We pretrain the MolCloze model via three tasks, which are Unordered Masked Language Modeling (UMLM), Replaced Masked Token Detection (RMTD), and Contrastive Energy-based Unmasked Token Clozing (CE-UTC). Then, we transfer the pre-trained model to a broad range of downstream molecular property prediction tasks via minor architecture modification. Extensive experiments demonstrate the generalizability of MolCloze by predicting a broad range of chemical properties which are related to drug discovery. We also observe significant performance boost on different downstream molecular property prediction datasets, achieving higher performance than the state-of-the-art baseline approaches and previous pre-training techniques developed for molecule data.

Accurately identifying and predicting molecular properties is a crucial task in molecular machine learning, and the key lies in how to extract effective molecular representations. Contrastive learning opens new avenues for representation learning, and a large amount of unlabeled data enables the model to generalize to the huge chemical space. However, existing contrastive learning-based models face two challenges: (i) existing methods destroy the original molecular environment and ignore chemical prior information, and (ii) there is a lack of a prior knowledge to guide the prediction of molecular properties. In this work, we propose a molecular property prediction framework called MolFCL, which consists of fragment-based contrastive learning and functional group-based prompt learning. Specifically, we introduced fragment-fragment interactions for the first time in the contrastive learning framework and designed a fragment-based augmented molecular graph that integrates the original chemical environment and fragment reactions. Furthermore, we proposed a novel functional group-based prompt learning during fine-tuning, which first incorporates functional group knowledge and the corresponding atomic signals, to improve molecular representation and provide interpretable analyses. The results show that MolFCL outperforms state-of-the-art baseline models on 23 molecular property prediction datasets. Moreover, visualizations show that MolFCL can learn to embed molecules into representations that can distinguish chemical properties. MolFCL can give higher weight to functional groups consistent with chemical knowledge during the prediction of molecular properties, which offers an interpretable ability of the model. Overall, MolFCL is a practically useful tool for molecular property prediction and assists drug scientists in designing drugs more effectively. MolFCL is available at https://github.com/tangxiangcsu/MolFCLSupplementary.

Graph-based molecular representation learning is essential for predicting molecular properties in drug discovery and materials science. Despite its importance, current approaches struggle with capturing the intricate molecular relationships and often rely on limited chemical knowledge during training. Multimodal fusion, which integrates information from graph and other data sources together, has emerged as a promising approach for enhancing molecular property prediction. However, existing studies explore only a narrow range of modalities, and the optimal integration stages for multimodal fusion remain largely unexplored. Furthermore, the reliance on auxiliary modalities poses challenges, as such data is often unavailable in downstream tasks. Here, we present MMFRL (Multimodal Fusion with Relational Learning), a framework designed to address these limitations by leveraging relational learning to enrich embedding initialization during multimodal pre-training. MMFRL enables downstream models to benefit from auxiliary modalities, even when these are absent during inference. We also systematically investigate modality fusion at early, intermediate, and late stages, elucidating their unique advantages and trade-offs. Using the MoleculeNet benchmarks, we demonstrate that MMFRL significantly outperforms existing methods with superior accuracy and robustness. Beyond predictive performance, MMFRL enhances explainability, offering valuable insights into chemical properties and highlighting its potential to transform real-world applications in drug discovery and materials science.

Scene graph generation denotes the process of parsing a visual input into a graph representation for downstream reasoning tasks. Most existing research models input scenes as flat scene graphs, capturing solely on horizontal relationships between two objects, while disregarding vertical dependencies. Such flat scene graphs merely describe direct relationships between objects, and cannot represent higher-level semantics. The lack of hierarchical relationships also limits the performance of scene graphs in downstream reasoning tasks. In this work, we propose hierarchical scene graph generation, a new problem task that requires the model to generate scene graphs with clear hierarchy. To achieve this goal, a hierarchical scene graph generation with coarse-to-fine reasoning framework is proposed. It contains three stages: first, summarize the main caption of the scene; second, describe the main visual elements related to the theme; and finally, add secondary elements with background. To facilitate training, a high-quality hierarchical scene graph dataset that comprises 40 K image-scene description pairs is constructed. Utilize rich knowledge and powerful multi-turn conversation capabilities of multi-modal large language models, the proposed framework can not only achieve better semantic understanding and relational modeling, but also seamlessly integrating its relational modeling capabilities to enhance visual-language tasks. Extensive experimental results demonstrate that our method constructs an effective scene representation, outperforming the state-of-the-art by 3.35% in recall while generating significantly fewer average triplets 8.7vs. 9.2. The results on downstream tasks also indicate that hierarchical scene graph significantly contributes to visual-language interactive reasoning.

Existing molecular property prediction methods based on deep learning ignore the generalization ability of the nonlinear representation of molecular features and the reasonable assignment of weights of molecular features, making it difficult to further improve the accuracy of molecular property prediction. To solve the above problems, an end-to-end double-head transformer neural network (DHTNN) is proposed in this paper for high-precision molecular property prediction. For the data distribution characteristics of the molecular dataset, DHTNN specially designs a new activation function, beaf, which can greatly improve the generalization ability of the nonlinear representation of molecular features. A residual network is introduced in the molecular encoding part to solve the gradient explosion problem and ensure that the model can converge quickly. The transformer based on double-head attention is used to extract molecular intrinsic detail features, and the weights are reasonably assigned for predicting molecular properties with high accuracy. Our model, which was tested on the MoleculeNet [1] benchmark dataset, showed significant performance improvements over other state-of-the-art methods.

MotivationAccurate molecular property prediction remains a central challenge in molecular machine learning, critically dependent on comprehensive molecular representation. Existing methods, however, encounter two major limitations: (i) single-modal learning approaches frequently experience representation bottlenecks, whereas multimodal methods often struggle to effectively leverage complementary information without redundancy across modalities; and (ii) conventional data augmentation techniques typically treat atoms as isolated units, neglecting intrinsic dependencies among atoms within molecular substructures.ResultsHere, we propose MolCL-SP, a substructure-aware multimodal contrastive learning framework specifically designed for molecular property prediction. Our approach integrates molecular representations derived from three complementary modalities using a Transformer-based encoder, followed by modality-specific reconstruction to organically align and fuse cross-modal information. We also introduce a novel substructure-based non-overlapping perturbation strategy for data augmentation, preserving interpretability and effectively enhancing inter-modal interactions. Extensive experimental evaluations demonstrate that MolCL-SP achieves state-of-the-art performance on benchmark datasets for both 2D and 3D molecular property predictions. Additionally, evaluations on drug–drug interaction prediction tasks highlight the model’s strong generalization capabilities. Visualization analyses further indicate that MolCL-SP effectively captures discriminative molecular embeddings even in task-agnostic contexts. Importantly, the model implicitly emphasizes chemically meaningful substructures associated with functional relevance, significantly enhancing interpretability.Availability and implementationCodes and materials are available at https://github.com/lylikeeMoon/MolCL-SP.

Gene regulatory networks (GRNs) are crucial for understanding gene regulation and cellular processes. Inferring GRNs helps uncover regulatory pathways, shedding light on the regulation and development of cellular processes. With the rise of high-throughput sequencing and advancements in computational technology, computational models have emerged as cost-effective alternatives to traditional experimental studies. Moreover, the surge in ChIP-seq data for TF-DNA binding has catalyzed the development of graph neural network (GNN)-based methods, greatly advancing GRN inference capabilities. However, most existing GNN-based methods suffer from the inability to capture long-distance structural semantic correlations due to transitive interactions. In this paper, we introduce a novel GNN-based model named Hierarchical Graph Transformer with Contrastive Learning for GRN (HGTCGRN) inference. HGTCGRN excels at capturing structural semantics using a hierarchical graph Transformer, which introduces a series of gene family nodes representing gene functions as virtual nodes to interact with nodes in the GRNS. These semantic-aware virtual-node embeddings are aggregated to produce node representations with varying emphasis. Additionally, we leverage gene ontology information to construct gene interaction networks for contrastive learning optimization of GRNs. Experimental results demonstrate that HGTCGRN achieves superior performance in GRN inference.

In recent years, machine learning has gained increasing traction in the study of molecules, enabling researchers to tackle challenging tasks including molecular property prediction and drug design.Consequently, there remains an open challenge to develop a neural network architecture that can make use of extensive amounts of unlabeled data for training while still providing competitive results in various molecular property prediction tasks. To address this challenge, we propose a Molecule Graph Contrastive Learning approach based on the Transformer framework (T-MGCL). Our approach involves expanding numerous unsupervised molecular graphs and using a contrast estimator to ensure consistency among various graph expansions of the same molecule. Transformer framework is employed to consider the distance between atoms and molecular graph attributes, thereby accounting for structural information that may be overlooked by traditional graph neural networks. Our experimental results demonstrate that the T-MGCL model outperforms other models in several molecular property prediction tasks. Additionally, we observe that the attention weight learned by T-MGCL can be easily explained from a chemical perspective.

Pre-trained language models with structured semantic knowledge have demonstrated remarkable performance in a variety of downstream natural language processing tasks. The typical methods of integrating knowledge are designing different pre-training tasks and training from scratch, which requires high-end hardware, massive storage resources, and long computing times. Prompt learning is an effective approach to tuning language models for specific tasks, and it can also be used to infuse knowledge. However, most prompt learning methods accept one token as the answer, instead of multiple tokens. To tackle this problem, we propose the long-answer prompt learning method (KLAPrompt), with three different long-answer strategies, to incorporate semantic knowledge into pre-trained language models, and we compare the performance of these three strategies through experiments. We also explore the effectiveness of the KLAPrompt method in the medical field. Additionally, we generate a word sense prediction dataset (WSP) based on the Xinhua Dictionary and a disease and category prediction dataset (DCP) based on MedicalKG. Experimental results show that discrete answers with the answer space partitioning strategy achieve the best results, and introducing structured semantic information can consistently improve language modeling and downstream tasks.

Molecular property prediction has become essential inacceleratingadvancements in drug discovery and materials science. Graph NeuralNetworks have recently demonstrated remarkable success in molecularrepresentation learning; however, their broader adoption is impededby two significant challenges: (1) data scarcity and constrained modelgeneralization due to the expensive and time-consuming task of acquiringlabeled data and (2) inadequate initial node and edge features thatfail to incorporate comprehensive chemical domain knowledge, notablyorbital information. To address these limitations, we introduce aKnowledge-Guided Graph (KGG) framework employingself-supervised learning to pretrain models using orbital-level featuresin order to mitigate reliance on extensive labeled data sets. In addition,we propose novel representations for atomic hybridization and bondtypes that explicitly consider orbital engagement. Our pretrainingstrategy is cost efficient, utilizing approximately 250,000 moleculesfrom the ZINC15 data set, in contrast to contemporary approaches thattypically require between two and ten million molecules, consequentlyreducing the risk of potential data contamination. Extensive evaluationson diverse downstream molecular property data sets demonstrate thatour method significantly outperforms state-of-the-art baselines. Complementaryanalyses, including t-SNE visualizations andcomparisons with traditional molecular fingerprints, further validatethe effectiveness and robustness of our proposed KGG approach. The key advantages of KGG are itsdata efficiency and architectural versatility, driven by orbital-informedrepresentations. By distilling essential chemical knowledge from modestcorpora, it avoids extensive pretraining and excels in low-data fine-tuning,providing a robust and chemically meaningful foundation for diverse GNN architectures.

The present state and challenges of active learning in drug discovery

Molecular property prediction (MPP) is vital in drug discovery and drug reposition. Deep learning-based MPP models capture molecular property-related features from various molecule representations. In this paper, we propose a molecule sequence embedding and prediction model facing with MPP task. We pre-trained a bi-directional encoder representations from Transformers (BERT) encoder to obtain the semantic representation of compound fingerprints, called Fingerprints-BERT (FP-BERT), in a self-supervised learning manner. Then, the encoded molecular representation by the FP-BERT is input to the convolutional neural network (CNN) to extract higher-level abstract features, and the predicted properties of the molecule are finally obtained through fully connected layer for distinct classification or regression MPP tasks. Comparison with the baselines shows that the proposed model achieves high prediction performance on all of the classification tasks and regression tasks.