M^2LLM: Multi-view Molecular Representation Learning with Large Language Models

Computational drug repositioning, through predicting drug-disease associations (DDA), offers significant potential for discovering new drug indications. Current methods incorporate graph neural networks (GNN) on drug-disease heterogeneous networks to predict DDAs, achieving notable performances compared to traditional machine learning and matrix factorization approaches. However, these methods depend heavily on network topology, hampered by incomplete and noisy network data, and overlook the wealth of biomedical knowledge available. Correspondingly, large language models (LLMs) excel in graph search and relational reasoning, which can possibly enhance the integration of comprehensive biomedical knowledge into drug and disease profiles. In this study, we first investigate the contribution of LLM-inferred knowledge representation in drug repositioning and DDA prediction. A zero-shot prompting template was designed for LLM to extract high-quality knowledge descriptions for drug and disease entities, followed by embedding generation from language models to transform the discrete text to continual numerical representation. Then, we proposed LLM-DDA with three different model architectures (LLM-DDANode Feat, LLM-DDADual GNN, LLM-DDAGNN-AE) to investigate the best fusion mode for LLM-based embeddings. Extensive experiments on four DDA benchmarks show that, LLM-DDAGNN-AE achieved the optimal performance compared to 11 baselines with the overall relative improvement in AUPR of 23.22%, F1-Score of 17.20%, and precision of 25.35%. Meanwhile, selected case studies of involving Prednisone and Allergic Rhinitis highlighted the model's capability to identify reliable DDAs and knowledge descriptions, supported by existing literature. This study showcases the utility of LLMs in drug repositioning with its generality and applicability in other biomedical relation prediction tasks.

Suicide is the fourth leading cause of death among people aged 15-29. More than 720, 000 people commit suicide every year. During the COVID-19 pandemic, we saw an increase in people seeking out mental health support on anonymous forums like Reddit. These anonymous forums allow people to express their suicidal ideation without judgment and give them a support structure that not everyone has. The aim of this project is to detect suicidal ideation using Reddit. In this work, we propose SIRGEL (Suicidal Ideation on Reddit using Graph Embeddings and LLMs), a dual-pipeline approach that combines large language model (LLM)- based annotation with both traditional text classification and graph neural network (GNN)-based modeling. The first step is to create a Reddit dataset for related subreddits. The second step is to annotate them as suicidal or not suicidal using LLMs. We also detail the creation of a new dataset made using Reddit posts and its benchmarking using baseline machine learning models. We model this problem using two approaches: traditional text classification and graph representation learning. Additionally, experiments with different types of Graph Neural Networks are performed with optimizations to improve and achieve efficient results

Rich textual and topological information of textual graphs need to be modeled in real-world applications such as webpages, e-commerce, and academic articles. Practitioners have been long following the path of adopting a shallow text encoder and a subsequent graph neural network (GNN) to solve this problem. In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs. Nevertheless, the efficiency of these methods poses a significant challenge. In this paper, we propose ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs with an LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable side structure, which significantly reduces the training complexity without impairing the joint model's capacity. Extensive experiments on textual graphs demonstrate our method's effectiveness by achieving the best model performance, meanwhile having the lowest training cost compared to previous methods. Moreover, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Specifically, caching accelerates ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster inference with a negligible performance drop (at maximum 1.17% relevant drop across 7 datasets). Our codes are available at: https://github.com/ZhuYun97/ENGINE.

Large Language Models (LLMs) became wide-spread during only the last couple of years and are used in almost every scientific and non-scientific domain. Understanding opportunities, applications and limitations of LLMs is crucial for a risk-free, effective and useful implementation of LLMs into scientific workflows. We demonstrate that their effectiveness is maximized not through autonomous operation but through careful integration with specialized tools and contextual knowledge bases. Using local deployments of modern LLMs (QWen-2.5-coder, LLaMA, Mistral) comes with a number of benefits in a scientific context. Our approach employs vector embeddings for enhanced context retention and metadata databases for structured data access, enabling guided, context-aware interactions with the LLM. Local deployments allow for improved data handling and privacy, improved cost management and a higher degree of customization. Energy consumption can more easily be observed and managed, which can be a crucial property of such a system, especially compared to the newest generation of LLMs, which have extensive power  (and cost) requirements.Opportunities and limitations are explored through two case studies: (1) an LLM-driven system that queries metadata databases to retrieve data from common open data sources and harmonizes patio-temporal subsets into data-cubes, and (2) a VBA-to-Python code translation project to preserve a legacy selection-system forest management software, which was developed in ACCESS / VBA over more than two decades. The LLM's translation process and reasoning are preserved in a vector database for consistent context maintenance and the original as well as the ‚new‘ code is searchable using the LLM to aid rebuilding a modernized software. Results suggest that this tool-augmented approach leads to a more reliable and maintainable solution compared to purely LLM-driven implementations, suggesting a new paradigm for integrating AI in scientific workflows where LLMs rather facilitate than replace domain-specific tools and human expertise.

Accurate prediction of molecular toxicity is vital for drug development. Most mainstream methods rely on fingerprints or graph-based feature extraction, the emergence of large language models (LLMs) offers new prospects for molecular representation learning in toxicity prediction. Although several studies attempt to leverage LLMs to integrate molecular sequence data for pretraining molecular representations, certain limitations remain. Current LLM-based approaches usually utilize solely on class embedding features, overlooking the rich information in sequence embedding. Moreover, integrating pre-trained molecular representations with multi-modal molecular data may further enhance performance in toxicity prediction. To address these challenges, we propose BFGTP, a BERT-guided two-stage molecular representation learning framework for toxicity prediction. Firstly, we design independent encoders for molecular descriptions of three modalities, where the fingerprint encoder with dual level attention mechanisms effectively integrates multi-category fingerprints. Then, the two-stage guide strategy is introduced to fully utilize the prior knowledge of LLMs, employing contrastive learning to align and fuse the tri-modal representations and knowledge distillation to align predicted value distributions. BFGTP ultimately combines fingerprint and graph representations to predict molecular toxicity. Experiments on seven toxicity datasets show that BFGTP outperforms baselines, achieving the highest AUC on five datasets and the best average performance across five evaluation metrics. Ablation studies, t-SNE visualization and case study confirm the effectiveness of BFGTP's components and its ability to capture meaningful molecular representations.

BackgroundIt has been validated that dementia can be alleviated by talk therapy. While general large language models (LLMs) have the potential for talk therapy and dementia treatment, customized LLMs resembling a mental‐health therapist can be more professional and powerful. We introduced an LLM solution designed for tailored mental health support for future dementia care. By employing GPT self‐generated data, encompassing a diverse range of mental health scenarios, we've significantly enhanced the model’s ability to understand and engage with complex emotional and psychological queries from patients. This advancement in LLMs paves the way for more efficient and effective delivery of talk therapy, shedding light for future dementia care and treatment.MethodWe fine‐tuned five open‐source LLMs to enhance their capabilities in the context of mental health counseling. The fine‐tuning process was conducted using the quantized Low‐Rank Adapters (QLoRA) method on an A40 GPU. We generated 10,000 question‐answer pairs as the training data using GPT‐3.5‐Turbo, designed to simulate interactions between mental health counselors and patients. This dataset encompassed a broad spectrum of mental health issues, including but not limited to depression, anxiety, and sleep disorders. Post fine‐tuning, the performance of these models in test data gathered from the Mental Health Forum and Quora (Fig. 2) was judged comprehensively by GPT‐4 Turbo. This evaluation was structured around six mental health metrics (Fig. 1) to ensure precise and relevant assessment in the context of mental health therapy.ResultThe top‐performing model, “Mixtral‐8×7B‐v0.1‐qlora,” achieved a remarkable average score of 7.83 (Tab. 1), highlighting its superior capability over the base model in handling complex mental health interactions. Other fine‐tuned models, such as “Zephyr‐7b‐gpt” and “Vicuna‐7b‐gpt,” also showcased exceptional performance over their base models, scoring 7.82 and 7.78 respectively. These results underscore the efficacy of fine‐tuning in enhancing the empathetic and contextual responsiveness of LLMs in mental health settings, particularly in dementia care.ConclusionThe study’s results showcased the effectiveness of fine‐tuned large language models (LLMs) suffixed by “QLoRA” in enhancing talk therapy. The leading model, “Mistral‐7B‐Instruct‐v0.2‐QLoRA,” demonstrated superior empathy and understanding over the base model in complex dementia interactions by leveraging domain‐specific data.

Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used.However, fine-tuning still remains crucial to further enhance their adaptability.Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality.We address this issue by introducing a prompt-based parameter-efficient finetuning (PEFT) approach.GNNAVI leverages insights into ICL's information flow dynamics, which indicates that label words act in prompts as anchors for information propagation.GN-NAVI employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN.Our experiments on text classification tasks with GPT-2 and Llama2 show GNNAVI surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters.We compare GNNAVI with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency.Our analysis reveals that GNNAVI enhances information flow and ensures a clear aggregation process. 1

Deep learning models have demonstrated their potential in learning effective molecular representations critical for drug property prediction and drug discovery. Despite significant advancements in leveraging multimodal drug molecule semantics, existing approaches often struggle with challenges such as low-quality data and structural complexity. Large language models (LLMs) excel in generating high-quality molecular representations due to their robust characterization capabilities. In this work, we introduce GICL, a cross-modal contrastive learning framework that integrates LLM-derived embeddings with molecular image representations. Specifically, LLMs extract feature representations from the SMILES strings of drug molecules, which are then contrasted with graphical representations of molecular images to achieve a holistic understanding of molecular features. Experimental results demonstrate that GICL achieves state-of-the-art performance on the ADMET task while offering interpretable insights into drug properties, thereby facilitating more efficient drug design and discovery.

Performance of Popular Large Language Models in Glaucoma Patient Education: a randomized controlled study

Countless disasters have resulted from climate change, causing severe damage to infrastructure and the economy. These disasters have significant societal impacts, necessitating mental health services for the millions affected. To prepare for and respond effectively to such events, it is crucial to understand people's emotions and the life incidents they experience both before and after a disaster strikes. In this case study, we mined the emotions and life incidents expressed on Twitter during the 2017 Hurricane Harvey. Between August 20 and August 30, we collected a dataset of approximately 400,000 public tweets related to the storm. Using a BERT-based model, we predicted the emotions associated with each tweet. Following emotion prediction, we applied various natural language processing techniques to analyze the content of the tweets, aiming to uncover the 'topics' discussed alongside specific 'emotions.' To efficiently identify these topics, we utilized the Latent Dirichlet Allocation (LDA) technique for topic modeling, which allowed us to bypass manual content analysis and extract meaningful patterns from the data. However, rather than stopping at topic identification, we further refined our analysis by integrating Graph Neural Networks (GNN) and Large Language Models (LLM). The GNN was employed to generate embeddings and construct a similarity graph of the tweets, which was then used to optimize clustering. This step ensured that related tweets were accurately grouped together based on their content and context. Subsequently, we used an LLM to automatically generate descriptive names for each event cluster. This advanced approach allowed us to identify and label life incidents automatically, providing a nuanced understanding of the key issues faced by the community during the hurricane. The results revealed significant emotional shifts as the hurricane reached its most devastating phase. These shifts coincided with pressing concerns and demands from the population, such as evacuation plans, animal safety, safety updates, public policy, help requests, flood impacts, and the spread of fake news. By combining GNN and LLM for automatic event name prediction, our study not only mapped out the emotions but also accurately categorized the specific life incidents that were most relevant during the disaster, offering critical insights for disaster preparedness and response strategies.

Identifying and Generating research papers' limitations in scientific articles will enhance the transparency and rigor of scientific research. In this work, we aim to automatically generate Limitations sections in scientific articles based on other key sections, including Abstract, Introduction, Methodology, Related Work, Experiment, and Conclusion. To identify the most relevant content, we apply a cosine similarity-based approach to select important sections for input. For generating limitations, we experiment with several large language models (LLMs), including BART, T5, Pegasus, GPT-3.5, GPT-4, and Gemini. Our findings show that GPT-3.5 with Retrieval-Augmented Generation (RAG) outperforms other models in accurately generating limitations. Building on this, we plan to incorporate a graph-based model using a graph neural network (GNN) that leverages both citation networks and thematic similarity to enhance the generation of limitations. This graph model will be integrated with the LLM and RAG system. Additionally, we intend to create a strong ground truth by incorporating both OpenReview and Limitations sections in the training data. For evaluation, we will apply PointWise similarity with interpretable metrics, utilizing the LLM as a judge approach to assess outcomes, supplemented by a human-in-the-loop approach to further refine results.

Graphs with abundant attributes are essential in modeling interconnected entities and enhancing predictions across various real-world applications. Traditional Graph Neural Networks (GNNs) often require re-training for different graph tasks and datasets. Although the emergence of Large Language Models (LLMs) has introduced new paradigms in natural language processing, their potential for generic graph mining-training a single model to simultaneously handle diverse tasks and datasets-remains under-explored. To this end, our novel framework ${\sf MuseGraph}$MuseGraph, seamlessly integrates the strengths of GNNs and LLMs into one foundation model for graph mining across tasks and datasets. This framework first features a compact graph description to encapsulate key graph information within language token limitations. Then, we propose a diverse instruction generation mechanism with Chain-of-Thought (CoT)-based instruction packages to distill the reasoning capabilities from advanced LLMs like GPT-4. Finally, we design a graph-aware instruction tuning strategy to facilitate mutual enhancement across multiple tasks and datasets while preventing catastrophic forgetting of LLMs' generative abilities. Our experimental results demonstrate significant improvements in five graph tasks and ten datasets, showcasing the potential of our ${\sf MuseGraph}$MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while improving the generation abilities of LLMs.

With the rise of large language models (LLM), there has been deemed a possible alternative to human participants in many scientific domains, including linguistic studies, the cloze study. Cloze probability is used to inform researchers as to how predictable a word is within a certain sentential context. It is a common tool in linguistic studies to understand language production and processing. Previous studies (e.g., Jacobs et al., 2022; Lopes Rego et al., 2024) have compared LLM performance with traditional cloze studies and their results are promising. Nonetheless, these studies were done in English. Hence, we would like to know LLM performance in the Portuguese language. Here, we compared results from a traditional cloze study with two LLM, such as: Grevásio (Santos et al., 2024) and Tucano (Corrêa et al., 2024) then performed a correlation analysis to investigate their performance. The results show a moderate and weak correlation between the cloze probability from human participants and the LLMs. These results highlight the gap between human performance and LLM, specifically in cloze probability.

A survey of low-bit large language models: Basics, systems, and algorithms.

BackgroundIn recent years, large language models (LLMs) have shown promise in various domains, notably in biomedical sciences. However, their real-world application is often limited by issues like erroneous outputs and hallucinatory responses.ResultsWe developed the knowledge graph–based thought (KGT) framework, an innovative solution that integrates LLMs with knowledge graphs (KGs) to improve their initial responses by utilizing verifiable information from KGs, thus significantly reducing factual errors in reasoning. The KGT framework demonstrates strong adaptability and performs well across various open-source LLMs. Notably, KGT can facilitate the discovery of new uses for existing drugs through potential drug–cancer associations and can assist in predicting resistance by analyzing relevant biomarkers and genetic mechanisms. To evaluate the knowledge graph question answering task within biomedicine, we utilize a pan-cancer knowledge graph to develop a pan-cancer question answering benchmark, named pan-cancer question answering.ConclusionsThe KGT framework substantially improves the accuracy and utility of LLMs in the biomedical field. This study serves as a proof of concept, demonstrating its exceptional performance in biomedical question answering.