GraFT: Infusing Pre-trained Transformers with Relational Structure for Time Series Forecasting

Large language models (LLMs) such as ChatGPT have shown significant potential in radiology. Their effectiveness often depends on prompt engineering, which optimizes the interaction with the chatbot for accurate results. Here, we highlight the critical role of prompt engineering in tailoring the LLMs' responses to specific medical tasks. Using a clinical case, we elucidate different prompting strategies to adapt the LLM ChatGPT using GPT4 to new tasks without additional training of the base model. These approaches range from precision prompts to advanced in-context methods such as few-shot and zero-shot learning. Additionally, the significance of embeddings, which serve as a data representation technique, is discussed. Prompt engineering substantially improved and focused the chatbot's output. Moreover, embedding of specialized knowledge allows for more transparent insight into the model's decision-making and thus enhances trust. Despite certain challenges, prompt engineering plays a pivotal role in harnessing the potential of LLMs for specialized tasks in the medical domain, particularly radiology. As LLMs continue to evolve, techniques like few-shot learning, zero-shot learning, and embedding-based retrieval mechanisms will become indispensable in delivering tailored outputs. · Large language models might impact radiological practice and decision-masking.. · However, implementation and performance are dependent on the assigned task.. · Optimization of prompting strategies can substantially improve model performance.. · Strategies for prompt engineering range from precision prompts to zero-shot learning.. · Russe MF, Reisert M, Bamberg F et al. Improving the use of LLMs in radiology through prompt engineering: from precision prompts to zero-shot learning . Fortschr Röntgenstr 2024; 196: 1166 - 1170.

Text-attributed heterogeneous graphs (TAHGs), characterized by nodes interconnected through diverse relationships and enriched with textual descriptions, are prevalent in numerous real-world applications. Recent advancements in integrating pre-trained language models (PLMs) and large language models (LLMs) with heterogeneous graph neural networks (HGNNs) have enhanced learning on TAHGs. However, the absence of standardized benchmark datasets tailored to TAHGs has impeded further progress. To bridge this gap, we propose the Text-attributed Heterogeneous Graphs Benchmark (THGB), a comprehensive collection of heterogeneous graphs from diverse domains, with each node enriched by relevant text attributes. Alongside dataset construction, we conduct extensive benchmark experiments using various graph learning methods, including GNN, PLM-GNN, and LLM-GNN approaches, for node classification and link prediction tasks. We evaluated model performance across supervised, few-shot, and zero-shot learning scenarios to assess their ability to leverage limited and unseen data. Our experiments highlight THGB's potential to improve the integration of heterogeneous structural and textual information. By providing curated datasets, robust evaluation protocols, and baseline implementations, THGB introduces a standardized benchmark and solid groundwork for TAHGs research.

We review current and emerging knowledge-informed and brain-inspired cognitive systems for realizing adversarial defenses, eXplainable Artificial Intelligence (XAI), and zero-shot or few-shot learning. Data-driven machine learning models have achieved remarkable performance and demonstrated capabilities surpassing humans in many applications. Yet, their inability to exploit domain knowledge leads to serious performance limitations in practical applications. In particular, deep learning systems are exposed to adversarial attacks, which can trick them into making glaringly incorrect decisions. Moreover, complex data-driven models typically lack interpretability or explainability, i.e., their decisions cannot be understood by human subjects. Furthermore, models are usually trained on standard datasets with a closed-world assumption. Hence, they struggle to generalize to unseen cases during inference in practical open-world environments, thus, raising the zero- or few-shot generalization problem. Although many conventional solutions exist, explicit domain knowledge, brain-inspired neural networks and cognitive architectures offer powerful new dimensions towards alleviating these problems. Prior knowledge is represented in appropriate forms like mathematical relations, logic rules, knowledge graphs, and large language models (LLMs). and incorporated in deep learning frameworks to improve performance. Brain-inspired cognition methods use computational models that mimic the human brain to enhance intelligent behavior in artificial agents and autonomous robots. Ultimately, these models achieve better explainability, higher adversarial robustness and data-efficient learning, and can, in turn, provide insights for cognitive science and neuroscience—that is, to deepen human understanding on how the brain works in general, and how it handles these problems.

A survey for large language models in biomedicine.

This paper investigates the use of ChatGPT, a large language model, for simplifying long sentences and nominal clusters in professional texts belonging to administrative and legal domains. We apply three prompt engineering techniques — zero-shot learning, few-shot learning, and Chain-of-Thought reasoning — to generate alternative sentences from a corpus of Italian texts. We evaluate the generated sentences using a survey with expert and non-expert readers of bureaucratic and legal Italian, focusing on ease of understanding, coherence, and preferences in rephrasing. Our results show that ChatGPT can effectively address the linguistic challenges outlined by UNI 11482:2013 Standard, and that complex prompting techniques yield better outcomes than simpler ones. We also discuss the implications of our findings for the optimization of text understanding and simplification using large language models.

Introduction/Background: In-hospital cardiac arrest (IHCA) is experienced by approximately 300,000 patients annually in the US. While individual care teams can readily identify IHCA at the bedside, the reporting of these events after the event often lacks consistency and poses a major challenge for hospitals across the US. Accurate and timely reporting of IHCA events is crucial for facilitating QI initiatives, such as CPR quality review, optimizing team performance, and benchmarking IHCA outcomes. Hypothesis: Large language models (LLMs) can be used to identify IHCA events at the patient encounter-level by analyzing targeted note types from the inpatient record with high performance. Methods: Adult (≥18 years old) inpatient encounters at the Hospital of the University of Pennsylvania from 06/2018 to 03/2022 with a reported clinical emergency were identified from a QI database. Discharge summaries and notes associated with the encounters of interest were extracted from the EHR and deidentified using a natural language processing tool. Notes were reviewed by research staff to ascertain true IHCA events. Positive IHCA was defined as the loss of pulses followed by the delivery of CPR. An LLM, GPT4 v. 32K-Chat, was employed on the Penn Medicine Microsoft Azure Databricks environment to compare zero- and few-shot prompting methods to identify IHCA. Model performance was measured using accuracy, precision, recall and F1 scores. Results: Manual chart review determined 96.8% and 3.2% of cases to be IHCA+ and IHCA-, respectively. A balanced pilot dataset of positive and negative cases was created and the accuracy values across strategically sampled note types was checked. Using zero-shot learning with Chain-of-Thought (COT) prompting, the LLM performed with an accuracy of 86%, precision of 77%, recall of 100%, and F1-score of 87%. Using three-shot learning with COT, the LLM performed with an accuracy of 90%, precision of 83%, recall of 100%, and F1-score of 91%. Conclusions: This study demonstrates the potential efficacy of leveraging LLMs to automatically classify IHCA from strategically sampled note types using both zero and few-shot learning approaches. In the future, we will implement temporal sampling of note types and refine model parameters such as output token size and temperature. These findings suggest that such technologies can be effective in real-world clinical settings, providing a scalable solution to improve patient outcomes and quality improvement efforts.

Sentiment analysis is a computational technique employed to extract and interpret subjective information from textual data. It involves the identification and classification of sentiments, opinions, and emotions expressed within the text. By analyzing linguistic cues, such as word choice, syntax, and sentiment lexicons, sentiment analysis can discern a range of emotions, from positive to negative, as well as more nuanced sentiments, such as anger, joy, or surprise. This powerful tool has the potential to unlock valuable insights from vast amounts of unstructured text data, which enables informed decision-making and effective communication in various domains, including education. Recent advances in sentiment analysis have leveraged the power of deep neural networks, particularly general-purpose Large Language Models (LLMs) trained on extensive labeled datasets. However, real-world applications frequently encounter challenges related to the availability of large, high-quality labeled data and the computational resources necessary for training such models. This research addresses these challenges by investigating effective strategies for utilizing LLMs in scenarios with limited data and computational resources. Specifically, this study explores three techniques: zero-shot learning, <i>N</i>-shot learning and fine-tuning. By evaluating these methods, this research aims to demonstrate the feasibility of employing general-purpose LLMs for sentiment analysis within educational contexts even when access to computational resources and labeled data is limited. The findings of this study reveal that different adaptation methods lead to significantly different LLM performance outcomes.

Game theory establishes a fundamental framework for analyzing strategic interactions among rational decision-makers. The rapid advancement of large language models (LLMs) has sparked extensive research exploring the intersection of these two fields. Specifically, game-theoretic methods are being applied to evaluate and enhance LLM capabilities, while LLMs themselves are reshaping classic game models. This paper presents a comprehensive survey of the intersection of these fields, exploring a bidirectional relationship from three perspectives: (1) Establishing standardized game-based benchmarks for evaluating LLM behavior; (2) Leveraging game-theoretic methods to improve LLM performance through algorithmic innovations; (3) Characterizing the societal impacts of LLMs through game modeling. Among these three aspects, we also highlight how the equilibrium analysis for traditional game models is impacted by LLMs' advanced language understanding, which in turn extends the study of game theory. Finally, we identify key challenges and future research directions, assessing their feasibility based on the current state of the field. By bridging theoretical rigor with emerging AI capabilities, this survey aims to foster interdisciplinary collaboration and drive progress in this evolving research area.

Psychometric and moral benchmarks are increasingly used to evaluate large language models (LLMs), aiming to measure their capabilities, surface implicit biases, and assess alignment with human values. However, interpreting LLM responses to these benchmarks is methodologically challenging, a nuance often overlooked in existing literature. We empirically demonstrate that LLM responses to a standard psychometric benchmark (generalized trust from the World Values Survey) correlate strongly with known survey results across language communities. Critically, we observe LLMs achieve this while explicitly referencing known survey results and the broader literature, even without direct prompting. We further show these correlations can be amplified or effectively eliminated by subtle changes in evaluation task design, revealing that replicating known results does not validate LLMs as naive subjects. Given LLMs' access to relevant literature, their ability to replicate known human behavior constitutes an invalid evaluation for assessing the suitability of large language models as naive subjects. Fascinating though it may be, this ability provides no evidence of generalizability to novel or out-of-sample behaviors. We discuss implications for alignment research and benchmarking practices.

Mental health diagnoses possess unique challenges that often lead to nuanced difficulties in managing an individual’s well-being and daily functioning. Self-report questionnaires are a common practice in clinical settings to help mitigate the challenges involved in mental health disorder screening. However, these questionnaires rely on an individual’s subjective response which can be influenced by various factors. Despite the advancements of Large Language Models (LLMs), quantifying self-reported experiences with natural language processing has resulted in imperfect accuracy. This project aims to demonstrate the effectiveness of zero-shot learning LLMs for screening and assessing item scales for depression using LLMs. The DAIC-WOZ is a publicly available mental health dataset that contains textual data from clinical interviews and self-report questionnaires with relevant mental health disorder labels. The RISEN prompt engineering framework was utilized to evaluate LLMs’ effectiveness in predicting depression symptoms based on individual PHQ-8 items. Various LLMs, including GPT models, Llama3_8B, Cohere, and Gemini were assessed based on performance. The GPT models, especially GPT-4o, were consistently better than other LLMs (Llama3_8B, Cohere, Gemini) across all eight items of the PHQ-8 scale in accuracy (M = 75.9%), and F1 score (0.74). GPT models were able to predict PHQ-8 items related to emotional and cognitive states. Llama 3_8B demonstrated superior detection of anhedonia-related symptoms and the Cohere LLM’s strength was identifying and predicting psychomotor activity symptoms. This study provides a novel outlook on the potential of LLMs for predicting self-reported questionnaire scores from textual interview data. The promising preliminary performance of the various models indicates there is potential that these models could effectively assist in the screening of depression. Further research is needed to establish a framework for which LLM can be used for specific mental health symptoms and other disorders. As well, analysis of additional datasets while fine-tuning models should be explored.

A review of research in the field of chemistry that incorporates machine learning is essential to identify recent developments and explore its potential applications. Published research articles provide an opportunity to analyze emerging research trends. The use of natural language processing (NLP) technology not only accelerates text data analysis but also enhances accuracy in understanding the content and context of scientific articles. Previously, trend analysis in ophthalmology research had been conducted using Zero-Shot Learning. In this study, an analysis of chemistry-related articles focusing on machine learning was carried out using a multi-turn prompting technique. The process began with data collection through web scraping of abstracts containing the keywords "machine learning" and "chemistry." The retrieved data was then tabulated and analyzed using a Large Language Model (LLM) with a Multi-Turn Prompting approach, where general prompts were initially used, followed by deeper exploration based on previous responses. Additionally, statistical descriptive analysis was performed using targeted prompts. Analysis of 200 article abstracts identified seven key terms related to the use of machine learning in chemistry: chemical (138 articles), protein (119 articles), drug (107 articles), structure (100 articles), molecular (96 articles), chemistry (91 articles), and quantum (84 articles). Furthermore, three dominant research topics were found in the intersection of chemistry and machine learning: protein and molecular structure, quantum chemistry, and drug discovery. The number of articles on machine learning in chemistry began to rise in 2012 and saw a significant increase in 2019. The findings suggest that there are still many opportunities for developing machine learning applications in chemistry, particularly in quantum chemistry. This field only began to gain attention in 2013, and the number of published articles remains relatively low each year, indicating that it is still in the early stages of exploration.

Creating biological knowledge bases and ontologies relies on time-consuming curation. Newly-emerging approaches driven by artificial intelligence and natural language processing can assist curators in populating these knowledge bases, but current approaches rely on extensive training data and are unable to populate arbitrarily complex nested knowledge schemas. We have developed Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES), a Knowledge Extraction approach that relies on the ability of Large Language Models (LLMs) to perform zero-shot learning (ZSL) and general-purpose query answering from flexible prompts and return information conforming to a schema. Given a user-defined schema and an input text, SPIRES recursively queries GPT-3+ to obtain responses matching the schema. SPIRES uses existing ontologies and vocabularies to provide identifiers for all matched elements. SPIRES may be applied to varied tasks, including extraction of cellular signaling pathways, disease treatments, drug mechanisms, and chemical to disease causation graphs. This approach offers easy customization, flexibility, and the ability to perform new tasks in the absence of any additional training data. SPIRES supports a strategy of leveraging the language interpreting capabilities of LLMs to assemble knowledge bases, assisting manual knowledge curation and acquisition while supporting validation with publicly-available databases and ontologies external to the LLM.

Patients with diabetes are at increased risk of comorbid depression or anxiety, complicating their management. This study evaluated the performance of large language models (LLMs) in detecting these symptoms from secure patient messages. We applied multiple approaches, including engineered prompts, systemic persona, temperature adjustments, and zero-shot and few-shot learning, to identify the best-performing model and enhance performance. Three out of five LLMs demonstrated excellent performance (over 90% of F-1 and accuracy), with Llama 3.1 405B achieving 93% in both F-1 and accuracy using a zero-shot approach. While LLMs showed promise in binary classification and handling complex metrics like Patient Health Questionnaire-4, inconsistencies in challenging cases warrant further real-life assessment. The findings highlight the potential of LLMs to assist in timely screening and referrals, providing valuable empirical knowledge for real-world triage systems that could improve mental health care for patients with chronic diseases.

Unleashing the potential of large language models (LLMs) in healthcare is a hot topic of research amongst computational scientists and bioinformatics researchers worldwide. Large language models have the potential to assist in various aspects of medicine given their capability to process complex concepts. The constant development of new treatment regimens is daunting for healthcare professionals and the introduction of these models allows for professionals to be updated with the latest advancements which improves patient care. CancerGPT [1] , an LLM, employs zero-shot learning to extract knowledge from medical literature and then use it to infer biological tasks. With approximately 124 million parameters, it matches the scale of the larger, fine-tuned GPT-3 model with around 175 million parameters. The observations from the study demonstrated that CancerGPT provided accurate responses for seven different biological tasks. However, the study also highlights the vulnerability of LLMs to hallucination. To address hallucination, which is incorrect and nonsensical content of large language models , our research seeks to address this limitation of LLM by seeking answers to the following questions. Can these models be trusted to responsibly recommend personalized treatment plans for cancer patients thereby improving their health outcomes? Can they serve as reliable clinical predictor tools for tailoring treatment strategies? To answer these questions, we curate a dataset consisting of treatment outcomes for monoclonal antibody treatment regimens in head and neck squamous cell carcinoma (HNSCC), which is the seventh most common cancer worldwide. The treatment regimens [2] include Cetuximab, Pembrolizumab, Durvalumab, and Nivolumab prescribed from the National Cancer Control Programme (NCCP) and The Irish Society of Medical Oncology (ISMO) which has a total of thirteen different drug regimens for HNSCC. The treatment outcomes are based on RECIST [3] score for each of the four drug regimens. Ten clinical trials for each of the individual treatment regimens were incorporated in our study. We augment prompt engineering process by providing contextual information such as drug regimens for targeted therapy, regimen doses, targeted oncogenes, and clinical outcomes from the clinical trials used in our study. We use one shot learning during prompt engineering phase. We test the LLM responses by providing contextual and non-contextual information. The LLM responses are evaluated using true positives and true negative scores on ten patient profiles of similar and different cancer type on the same drug regimen. Preliminary results indicate that contextual information significantly reduces LLM’s from hallucinating. To make the models reliable and trustworthy our study encompasses age, gender and genomic profile of patients. By overcoming hallucination and incorporating multimodal features, we demonstrate that LLMs holds the potential to recommend personalised treatments for HNSCC in Irish patients.

PurposeCurrent benchmarks evaluating large language models (LLMs) in ophthalmology are narrow and disproportionately prioritize accuracy. We introduce BEnchmarking LLMs for Ophthalmology (BELO), a standardized evaluation benchmark developed through multiple rounds of expert checking by 13 ophthalmologists. BEnchmarking LLMs for Ophthalmology assesses ophthalmology-related knowledge and reasoning quality.SubjectsThis study did not involve human participation.DesignCross-sectional study.MethodsUsing keyword matching and a fine-tuned PubMed Bidirectional Encoder Representations from Transformers model, we curated ophthalmology-specific multiple-choice questions (MCQs) from diverse medical data sets (Basic and Clinical Science Course [BCSC], Multi-Subject Multi-Choice Dataset for Medical domain [MedMCQA], Medical Question Answering [MedQA], Biomedical Semantic Indexing and Question Answering [BioASQ], and PubMed Question Answering [PubMedQA]). The data set underwent multiple rounds of expert checking. Duplicate and substandard questions were systematically removed. Ten ophthalmologists refined the explanations of each MCQ's correct answer. This was further adjudicated by 3 senior ophthalmologists. To illustrate BELO's utility, we evaluated 8 LLMs (OpenAI o1, o3-mini, GPT-5, GPT-4o, DeepSeek-R1, MedGemma-4B, Llama-3-8B, and Gemini 1.5 Pro).Main Outcome MeasuresThe 8 LLMs were evaluated in terms using accuracy, macro-F1, and 5 text-generation metrics (Recall-Oriented Understudy for Gisting Evaluation, BERTScore, BARTScore, Metric for Evaluation of Translation with Explicit Ordering, and AlignScore). In a further evaluation involving human experts, 2 ophthalmologists qualitatively reviewed 50 randomly selected outputs for accuracy, comprehensiveness, and completeness.ResultsBEnchmarking LLMs for Ophthalmology consists of 900 high-quality, expert-reviewed questions aggregated from 5 sources: BCSC (260), BioASQ (10), MedMCQA (572), MedQA (40), and PubMedQA (18). To demonstrate BELO's utility, we conducted a series of benchmarking exercises. In the quantitative evaluation, GPT-5 achieved the highest accuracy (0.90, 95% confidence interval [CI]: 0.89–0.92) and macro-F1 score (0.91, 95% CI: 0.89–0.93). On the other hand, the models' performance on text-generation metrics varied and were generally suboptimal, with scores ranging from 20.4 to 72.0 (out of 100, excluding the BARTScore metric), indicating room for improvement in clinical reasoning. In expert evaluations, GPT-4o was rated highest for accuracy and readability, while Gemini 1.5 Pro scored highest for completeness. A public leaderboard has been established to promote transparent evaluation and reporting. Importantly, the BELO data set will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models.ConclusionsBEnchmarking LLMs for Ophthalmology provides a robust clinically relevant benchmark for evaluating both the accuracy and reasoning capabilities of current and emerging LLMs in ophthalmology. Future BELO benchmarking efforts will expand to include vision-based question answering and clinical scenario management tasks.Financial Disclosure(s)The authors have no proprietary or commercial interest in any materials discussed in this article.