Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms

Knowledge graph reasoning (KGR) is the task of inferring new knowledge by performing logical deductions on knowledge graphs. Recently, large language models (LLMs) have demonstrated remarkable performance in complex reasoning tasks. Despite promising success, current LLM-based KGR methods still face two critical limitations. First, existing methods often extract reasoning paths indiscriminately, without assessing their different importance, which may introduce irrelevant noise that misleads LLMs. Second, while some methods leverage LLMs to dynamically explore potential reasoning paths, they require high retrieval demands and frequent LLM calls. To address these limitations, we propose PathMind, a novel framework designed to enhance faithful and interpretable reasoning by selectively guiding LLMs with important reasoning paths. Specifically, PathMind follows a "Retrieve-Prioritize-Reason" paradigm. First, it retrieves a query subgraph from KG through the retrieval module. Next, it introduces a path prioritization mechanism that identifies important reasoning paths using a semantic-aware path priority function, which simultaneously considers the accumulative cost and the estimated future cost for reaching the target. Finally, PathMind generates accurate and logically consistent responses via a dual-phase training strategy, including task-specific instruction tuning and path-wise preference alignment. Extensive experiments on benchmark datasets demonstrate that PathMind consistently outperforms competitive baselines, particularly on complex reasoning tasks with fewer input tokens, by identifying essential reasoning paths.

This paper examines the potential of integrating Large Language Models (LLMs) with knowledge bases to imrpove the accuracy and reliability of their responses. The advantages of such a combination are evaluated, particularly in reducing the risk of hallucinations – the phenomenon where models generate erroneous or fabricated information. Various methodologies for combining LLMs with knowledge bases are analyzed, along with their respective advantages and limitations. The prospects and challenges of implementing this technology in diverse fields—such as information retrieval, decision support, and automated content creation—are discussed. The paper presents an overview of the current state of research in this domain and delineates directions for future investigation.The integration of LLMs with knowledge bases represents a significant advancement in artificial intelligence, aiming one of the key concerns regarding LLMs—their tendency to generate inaccurate or fabricated information, commonly referred to as hallucinations. This approach leverages the vast language understanding and generation capabilities of LLMs while grounding their outputs in structured and verified information from knowledge bases. The synergy between these two technologies has the potential to significantly enhance the reliability and factual accuracy of AI-generated responses across a wide range of applications. The methodologies for combining LLMs with knowledge bases differ in their implementation and effectiveness. Some approaches involve pre-training LLMs on curated knowledge bases, while others reference knowledge bases externally during the inference process. Each method presents its own set of advantages and challenges, such as balancing computational efficiency against accuracy and maintaining the fluency of LLM outputs while adhering strictly to factual information. The application of this integrated technology extends beyond mere information retrieval, showing promise in complex decision support systems, automated content creation for specialized domains, and contributing to the advancement of explainable AI by providing traceable sources for generated information. As research in this area progresses, it is expected to open new avenues for developing more trustworthy and capable AI systems across various industries and academic disciplines.

Trained on vast corpora, Large Language Models (LLMs) have the potential to encode visualization design knowledge and best practices. However, if they fail to do so, they might provide unreliable visualization recommendations. What visualization design preferences, then, have LLMs learned? We contribute DracoGPT, a method for extracting, modeling, and assessing visualization design preferences from LLMs. To assess varied tasks, we develop two pipelines-DracoGPT-Rank and DracoGPT-Recommend-to model LLMs prompted to either rank or recommend visual encoding specifications. We use Draco as a shared knowledge base in which to represent LLM design preferences and compare them to best practices from empirical research. We demonstrate that DracoGPT can accurately model the preferences expressed by LLMs, enabling analysis in terms of Draco design constraints. Across a suite of backing LLMs, we find that DracoGPT-Rank and DracoGPT-Recommend moderately agree with each other, but both substantially diverge from guidelines drawn from human subjects experiments. Future work can build on our approach to expand Draco's knowledge base to model a richer set of preferences and to provide a robust and cost-effective stand-in for LLMs.

<sec> <title>BACKGROUND</title> Substance use disorder (SUD) affects more than 48 million people in the US, but access to care is often constrained by factors such as stigma, financial barriers, and limited service availability. While digital health interventions have begun to mitigate these challenges, large language models (LLMs) such as GPT introduce new possibilities for scalable SUD recovery support. However, empirical evidence regarding LLM use in SUD contexts re-mains fragmented and unevenly evaluated. A scoping review is needed to synthesize cur-rent approaches, surface methodological and safety gaps, and inform future research. </sec> <sec> <title>OBJECTIVE</title> This scoping review synthesizes current evidence on the use of LLMs in SUD and addiction contexts. It examines how LLM-based tools are being applied across the full spectrum of care, identifies common tasks and data sources, evaluates risks and safety concerns, and maps early evidence of effectiveness and gaps that inform future research. </sec> <sec> <title>METHODS</title> We conducted a scoping review in accordance with PRISMA-ScR guidelines. We searched medical and technical databases for studies that reported empirical findings on LLMs in drug-related SUD contexts. Of 675 records identified, 492 were screened at the title and abstract level, 60 underwent full-text review, and 12 studies met inclusion criteria. </sec> <sec> <title>RESULTS</title> All included studies were published in 2024 or 2025, indicating a very recent and rapidly evolving evidence base. Most studies used public social media or online forums (8 of 12), while 3 studies used clinical data such as EHR, and 1 study used multimodal data that combined sensors, images, and chat logs. GPT-family models were the most frequently used (10 of 12), often evaluated alongside Llama or Mistral models. Across studies, LLM applications clustered into four task categories: text classification and extraction, response generation, conversational decision support, and predictive modeling using LLM-derived features. Classification and extraction tasks were most common and often showed im-proved performance compared with traditional machine learning or rule-based baselines. While general-purpose LLMs were prone to generating stigmatizing, biased, or clinically inappropriate content in SUD-related scenarios, it was also reported that targeted prompt engineering substantially reduced stigmatizing language and improved perceived ade-quacy and appropriateness of responses. However, safety evaluations remained limited in scope and depth. </sec> <sec> <title>CONCLUSIONS</title> LLM-based tools show emerging potential to support SUD-related research and care, particularly scalable classification and information extraction, conversational screening, and feature derivation from unstructured text or speech. Yet current work is predominantly preclinical, relying on public datasets and simulated prompts, with limited evidence on safety, clinical effectiveness, or real-world implementation in recovery settings. Responsible use in SUD contexts will require a safety-first development approach that integrates human-in-the-loop validation, systematic prompt engineering, and design constraints such as curated response libraries. It will also require efforts to strengthen LLM literacy among clinicians, peer specialists, and people in recovery. </sec>

Large language models (LLMs) have demonstrated logical reasoning abilities, but their inferences remain non-traceable and lack formal guarantees. We introduce eXa-LM, a controlled natural language (CNL) interface between LLMs and first-order logic solvers. Based on a Controlled Natural Language, our approach aims to create an explicit, verifiable, and interpretable bridge between text and formal logic. It relies on three main components: (1) a reformulation prompt that constrains the LLM to produce a set of facts and rules in CNL, (2) the semantic analyzer eXaSem translating this CNL into a Prolog program made of extended Horn clauses, and (3) the logic engine eXaLog, which integrates a second-order meta-interpreter capable of inferring ontological properties.We evaluate eXa-LM on three standard benchmarks—PrOntoQA, ProofWriter and FOLIO—comparing it to GPT-4o baselines including Standard prompting, Chain-of-Thought, Logic-LM, LINC, and LLM-TP. Results show that eXa-LM matches or exceeds recent neuro-symbolic systems while providing full traceability of reasoning and intrinsic explainability. On FOLIO, eXa-LM achieves 92.9% accuracy, a +5.5 point gain over LLM-TP, the strongest competing GPT-4o-based method in our comparison.This approach demonstrates the feasibility of a transparent neuro-symbolic reasoning pipeline in which LLMs produce not direct inferences but formally controlled linguistic representations. eXa-LM opens the way to neuro-symbolic architectures that are safer, verifiable and extensible, ultimately integrating hypothetical, abductive or inductive reasoning. Program and data will be made publicly available upon publication.

Knowledge based question answering (KBQA) is a complex task for natural language understanding. Many KBQA approaches have been proposed in recent years, and most of them are trained based on labeled reasoning path. This hinders the system's performance as many correct reasoning paths are not labeled as ground truth, and thus they cannot be learned. In this paper, we introduce a new concept of KBQA system which can leverage multiple reasoning paths' information and only requires labeled answer as supervision. We name it as \textbfMutliple \textbfReasoning \textbfPaths KB\textbfQA System (MRP-QA). We conduct experiments on several benchmark datasets containing both single-hop simple questions as well as muti-hop complex questions, including WebQuestionSP (WQSP), ComplexWebQuestion-1.1 (CWQ), and PathQuestion-Large (PQL), and demonstrate strong performance.

Large Language Models (LLMs) became wide-spread during only the last couple of years and are used in almost every scientific and non-scientific domain. Understanding opportunities, applications and limitations of LLMs is crucial for a risk-free, effective and useful implementation of LLMs into scientific workflows. We demonstrate that their effectiveness is maximized not through autonomous operation but through careful integration with specialized tools and contextual knowledge bases.&amp;#160;Using local deployments of modern LLMs (QWen-2.5-coder, LLaMA, Mistral) comes with a number of benefits in a scientific context. Our approach employs vector embeddings for enhanced context retention and metadata databases for structured data access, enabling guided, context-aware interactions with the LLM. Local deployments allow for improved data handling and privacy, improved cost management and a higher degree of customization. Energy consumption can more easily be observed and managed, which can be a crucial property of such a system, especially compared to the newest generation of LLMs, which have extensive power&amp;#160; (and cost) requirements.Opportunities and limitations are explored through two case studies: (1) an LLM-driven system that queries metadata databases to retrieve data from common open data sources and harmonizes patio-temporal subsets into data-cubes, and (2) a VBA-to-Python code translation project to preserve a legacy selection-system forest management software, which was developed in ACCESS / VBA over more than two decades. The LLM's translation process and reasoning are preserved in a vector database for consistent context maintenance and the original as well as the &amp;#8218;new&amp;#8216; code is searchable using the LLM to aid rebuilding a modernized software.&amp;#160;Results suggest that this tool-augmented approach leads to a more reliable and maintainable solution compared to purely LLM-driven implementations, suggesting a new paradigm for integrating AI in scientific workflows where LLMs rather facilitate than replace domain-specific tools and human expertise.

The emergence of large language models (LLMs) has provided robust support for application tasks across various domains, such as name entity recognition (NER) in the general domain. However, due to the particularity of the medical domain, the research on understanding and improving the effectiveness of LLMs on biomedical named entity recognition (BNER) tasks remains relatively limited, especially in the context of Chinese text. In this study, we extensively evaluate several typical LLMs, including ChatGLM2-6B, GLM-130B, GPT-3.5, and GPT-4, on the Chinese BNER task by leveraging a real-world Chinese electronic medical record (EMR) dataset and a public dataset. The experimental results demonstrate the promising yet limited performance of LLMs with zero-shot and few-shot prompt designs for Chinese BNER tasks. More importantly, instruction fine-tuning significantly enhances the performance of LLMs. The fine-tuned offline ChatGLM2-6B surpassed the performance of the task-specific model BiLSTM+CRF (BC) on the real-world dataset. The best fine-tuned model, GPT-3.5, outperforms all other LLMs on the publicly available CCKS2017 dataset, even surpassing half of the baselines; however, it still remains challenging for it to surpass the state-of-the-art task-specific models, i.e., Dictionary-guided Attention Network (DGAN). To our knowledge, this study is the first attempt to evaluate the performance of LLMs on Chinese BNER tasks, which emphasizes the prospective and transformative implications of utilizing LLMs on Chinese BNER tasks. Furthermore, we summarize our findings into a set of actionable guidelines for future researchers on how to effectively leverage LLMs to become experts in specific tasks.

Structure reasoning is a fundamental capability of large language models (LLMs), enabling them to reason about structured commonsense and answer multi-hop questions. However, existing benchmarks for structure reasoning mainly focus on horizontal and coordinate structures (e.g. graphs), overlooking the hierarchical relationships within them. Hierarchical structure reasoning is crucial for human cognition, particularly in memory organization and problem-solving. It also plays a key role in various real-world tasks, such as information extraction and decision-making. To address this gap, we propose HiBench, the first framework designed to systematically benchmark the hierarchical reasoning capabilities of LLMs from initial structure generation to final proficiency assessment. It encompasses six representative scenarios, covering both fundamental and practical aspects, and consists of 30 tasks with varying hierarchical complexity, totaling 39,519 queries. To evaluate LLMs comprehensively, we develop five capability dimensions that depict different facets of hierarchical structure understanding. Through extensive evaluation of 20 LLMs from 10 model families, we reveal key insights into their capabilities and limitations: 1) existing LLMs show proficiency in basic hierarchical reasoning tasks; 2) they still struggle with more complex structures and implicit hierarchical representations, especially in structural modification and textual reasoning. Based on these findings, we create a small yet well-designed instruction dataset, which enhances LLMs' performance on HiBench by an average of 88.84% (Llama-3.1-8B) and 31.38% (Qwen2.5-7B) across all tasks. The HiBench dataset and toolkit are available at https://github.com/jzzzzh/HiBench to encourage evaluation.

We propose Contextual History for Adaptive and Simple Exploitation (CHASE), a novel multi-turn method for Large Language Model (LLM) jailbreaking. Rather than directly attack an LLM that may be difficult to jailbreak, CHASE first collects jailbroken histories from an easy-to-jailbreak LLM and then transfers them to the target LLM. Through this history transfer process, CHASE misleads the target LLM into thinking that it is responsible for producing the jailbroken histories and increases the chances of successful jailbreaking by prompting it to continue the conversation. Extensive evaluations on mainstream LLMs show that CHASE consistently achieves higher attack success rates and demands fewer computational resources compared to existing methods.

As large language models (LLMs) achieve remarkable success in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and are being actively explored currently. In this article, we focus on adapting and enhancing large language models for recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation realms, i.e., LLMs fail to effectively extract useful information from a pure textual context of long user behavior sequence, even if the length of context is well below the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely, R etrieval- e nhanced L arge La nguage models Plus (ReLLaX), which provides full-stack optimization from three perspectives, i.e., data, prompt, and parameter. For data-level enhancement, we design semantic user behavior retrieval (SUBR) to reduce the heterogeneity of the behavior sequence, thus lowering the difficulty for LLMs to extract the essential information from user behavior sequences. Although SUBR can improve the data quality, further increase in the sequence length will still raise its heterogeneity to a level where LLMs can no longer comprehend it. Hence, we further propose to perform prompt-level and parameter-level enhancement, with the integration of conventional recommendation models (CRMs). As for prompt-level enhancement, we apply soft prompt augmentation (SPA) to explicitly inject collaborative knowledge from CRMs into the prompt. The item representations of LLMs are thus more aligned with recommendation, helping LLMs better explore the item relationships in the sequence and facilitating comprehension. Finally, for parameter-level enhancement, we propose component fully-interactive LoRA (CFLoRA). By enabling sufficient interaction between the LoRA atom components, the expressive ability of LoRA is extended, making the parameters effectively capture more sequence information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e., from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments are conducted on three real-world public datasets to demonstrate the superiority of ReLLaX compared with existing baseline models, as well as its capability to alleviate lifelong sequential behavior incomprehension. Our code is available. 1

Large Language Models (LLMs) can be used as repositories of biological and chemical information to generate pharmacological lead compounds. However, for LLMs to focus on specific drug targets typically requires experimentation with progressively more refined prompts. Results thus become dependent not just on what is known about the target, but also on what is known about the prompt- engineering. In this paper, we separate the prompt into domain-constraints that can be written in a standard logical form and a simple text-based query. We investigate whether LLMs can be guided, not by refining prompts manually, but by refining the logical component automatically, keeping the query unchanged. We describe an iterative procedure LMLF (“Language Model with Logical Feedback”) in which the constraints are progressively refined using a logical notion of generalisation. On any iteration, newly generated instances are verified against the constraint, providing "logical-feedback" for the next iteration's refinement of the constraints. We evaluate LMLF using two well-known targets (inhibition of the Janus Kinase 2; and Dopamine Receptor D2); and two different LLMs (GPT-3 and PaLM). We show that LMLF, starting with the same logical constraints and query text, can be used to guide both LLMs to generate potential leads. We find: (a) Binding affinities of LMLF-generated molecules are skewed towards higher binding affinities than those from existing baselines; (b) LMLF results in generating molecules that are skewed towards higher binding affinities than without logical feedback; (c) Assessment by a computational chemist suggests that LMLF generated compounds may be novel inhibitors. These findings suggest that LLMs with logical feedback may provide a mechanism for generating new leads without requiring the domain-specialist to acquire sophisticated skills in prompt-engineering.

In this exploratory paper, I consider Ezumezu, a formal non-Western logic invented by Jonathan O. Chimakonam, and I take OpenAI’s GPT-5 as a broad paradigm of modern Large Language Model (LLM) capabilities. I examine the three supplementary laws of thought in Ezumezu logic, and find that GPT-5 is capable of reasoning only in alignment with the third supplementary law, Onona-etiti. In the case of Njikoka, I find that GPT-5 is entirely incapable of reasoning in accordance with the principle, and in the case of Nmekoka, it is capable of reasoning in accordance with the principle (curiously) only when prompted in the English language, and not when prompted in Igbo. I take a brief survey of the field to show that LLMs are not capable of logical reasoning, and frame this inability in terms of Raymond Cattell’s distinction between crystallised intelligence and fluid intelligence, showing that LLMs are highly proficient at the former, yet poor at the latter (a finding atypical of human cognitive patterns). I show that, therefore, in order to create a non-Western LLM that can effectively reason in Ezumezu logic, a neuro-symbolic framework is required, and I suggest the use of external symbolic executors as a good potential architecture. I note that such a future project would represent the first attempt at an LLM-based system designed specifically to reason in a non-Western logic.

Mathematical Problem Solving in Arabic: Assessing Large Language Models

Large Language Models (LLMs) have the potential to improve the quality of information services in higher education environments through responsive and natural interactions. However, LLMs are prone to generating answers that are not supported by valid knowledge sources due to knowledge cut-off limitations. This study implements Retrieval-Augmented Generation (RAG) on LLMs to build an information service chatbot for the Universitas Sains dan Teknologi Indonesia (USTI). RAG is built using a hybrid retrieval mechanism that combines dense retrieval (FAISS) and sparse retrieval (BM25) through Reciprocal Rank Fusion (RRF) and is equipped with cross-encoder reranking. The knowledge base is compiled from official and public documents obtained through the USTI website. The evaluation was conducted using 13 test queries by comparing several configurations to analyze the contribution of each component. The evaluation results show that the hybrid retrieval configuration produces the best retrieval performance with Precision@3 of 71.7%, Recall@3 of 87.5%, and NDCG@3 of 96.3%. In addition, the application of RAG improved the quality of answers compared to LLM without retrieval, as shown by an increase in BERTScore-F1 from 84.8% to 89.4% and a faithfulness score of 88.8%. These findings indicate that RAG integration improves the relevance of LLM answers to source documents, with the hybrid configuration providing an optimal balance between retrieval quality and faithfulness.