Chain-of-Thought-Based Knowledge Extraction from Heterogeneous Infrastructure Database for Integrated Transportation Asset Management

This paper investigates the impact of knowledge extraction methods on the effectiveness of RAG (Retrieval-Augmented Generation) systems that utilize knowledge graphs. It highlights that the quality of the knowledge graph, which is formed using various knowledge extraction methods, is crucial for overcoming limitations of large language models (LLMs), such as “hallucinations”. The paper analyzes the architectures of LightRAG and GraphRAG, emphasizing that the selection of an optimal knowledge extraction strategy depends on specific tasks and the subject area. LLMs have advanced significantly, but they have limitations, including generating factually incorrect information (“hallucinations”) and possessing “outdated knowledge”. RAG systems were proposed to address these issues by combining LLMs with external knowledge bases. This approach reduces hallucinations, ensures factual accuracy, solves the problem of outdated knowledge, and increases transparency. Knowledge graphs are powerful tools for structuring information, consisting of entities (nodes) and relations (edges). They enhance RAG systems by enabling more precise and contextually grounded retrieval compared to keyword-based searches. The quality of a knowledge graph depends on the knowledge extraction methods used, which include named entity recognition (NER), relation extraction (RE), entity linking (EL), and event extraction. Different methods, such as rule-based, classical machine learning, and deep learning approaches, have varying trade-offs in terms of accuracy, completeness, and scalability. Entity linking and knowledge graph completion are also crucial for accuracy and richness. LightRAG and GraphRAG are two main graph-based RAG systems. LightRAG uses the knowledge graph as a quick reference, requiring high precision in knowledge extraction to avoid noise degradation. GraphRAG uses the knowledge graph as a domain model, where completeness of extracted knowledge is more critical, though systematic errors are still harmful. Both systems rely on LLMs for knowledge extraction, which makes them dependent on the LLM’s quality and the size of document fragments processed. The theoretical analysis confirms that the effectiveness of RAG systems is critically dependent on knowledge extraction methods. The quality, completeness, and accuracy of the knowledge graph directly influence the RAG system’s ability to provide relevant, accurate, and truthful answers. Different RAG system architectures like LightRAG and GraphRAG have distinct requirements for knowledge graph characteristics, prioritizing either accuracy or completeness in knowledge extraction.

The use of information systems in transport is crucial for the construction, maintenance, formation, management and use of different types of transport services. They make it possible to achieve a comprehensive coverage of the different modes of transport, based on current regulatory and legal requirements in terms of technology, and financial considerations. This opens the possibility to manage various activities related to the optimization of travel and their quality, as well as to collect data in order to optimize, manage and minimize transport costs in road, water, air and rail transport systems. In this sense, an information system is one that, in the form of interconnected components, covers all forms of collecting, storing, retrieving, processing and providing information about the organization concerned. For this reason, the tendency in their development and implementation determines an aspiration for presentation at a new higher level capable of providing increased efficiency, precision and validity of management decisions. In this respect, the present paper examines the creation of a mathematical model allowing the intensification of rail passenger transport by introducing their baseline indicators in an information system, which is a basic trend for European countries. This model was used to create an information system allowing to solve a two-hierarchical optimization problem for train management between Sofia and Varna.

The exponential growth of digital content has generated massive textual datasets, necessitating the use of advanced analytical approaches. Large Language Models (LLMs) have emerged as tools that are capable of processing and extracting insights from massive unstructured textual datasets. However, how to leverage LLMs for text analytics Information Systems (IS) research is currently unclear. To assist the IS community in understanding how to operationalize LLMs, we propose a Text Analytics for Information Systems Research (TAISR) framework. Our proposed framework provides detailed recommendations grounded in IS and LLM literature on how to conduct meaningful text analytics IS research for design science, behavioral, and econometric streams. We conducted three business intelligence case studies using our TAISR framework to demonstrate its application in several IS research contexts. We also outline the potential challenges and limitations of adopting LLMs for IS. By offering a systematic approach and evidence of its utility, our TAISR framework contributes to future IS research streams looking to incorporate powerful LLMs for text analytics.

IntroductionRecent artificial intelligence (AI) advances can generate human-like responses to a wide range of queries, making them a useful tool for healthcare applications. Therefore, the potential use of large language models (LLMs) in controlled environments regarding efficacy, reproducibility, and operability will be of paramount interest.ObjectiveWe investigated if and how GPT 3.5 and GPT 4 models can be directly used as a part of a GxP validated system and compared the performance of externally hosted GPT 3.5 and GPT 4 against LLMs, which can be hosted internally. We explored zero-shot LLM performance for named entity recognition (NER) and relation extraction tasks, investigated which LLM has the best zero-shot performance to be used potentially for generating training data proposals, evaluated the LLM performance of seven entities for medical NER in zero-shot experiments, selected one model for further performance improvement (few-shot and fine-tuning: Zephyr-7b-beta), and investigated how smaller open-source LLMs perform in contrast to GPT models and to a small fine-tuned T5 Base.MethodsWe performed reproducibility experiments to evaluate if LLMs can be used in controlled environments and utilized guided generation to use the same prompt across multiple models. Few-shot learning and quantized low rank adapter (QLoRA) fine-tuning were applied to further improve LLM performance.Results and ConclusionWe demonstrated that zero-shot GPT 4 performance is comparable with a fine-tuned T5, and Zephyr performed better than zero-shot GPT 3.5, but the recognition of product combinations such as product event combination was significantly better by using a fine-tuned T5. Although Open AI launched recently GPT versions to improve the generation of consistent output, both GPT variants failed to demonstrate reproducible results. The lack of reproducibility together with limitations of external hosted systems to keep validated systems in a state of control may affect the use of closed and proprietary models in regulated environments. However, due to the good NER performance, we recommend using GPT for creating annotation proposals for training data as a basis for fine-tuning.

This paper describes the KnowledgeHub tool, a scientific literature Information Extraction (IE) and Question Answering (QA) pipeline. This is achieved by supporting the ingestion of PDF documents that are converted to text and structured representations. An ontology can then be constructed where a user defines the types of entities and relationships they want to capture. A browser-based annotation tool enables annotating the contents of the PDF documents according to the ontology. Named Entity Recognition (NER) and Relation Classification (RC) models can be trained on the resulting annotations and can be used to annotate the unannotated portion of the documents. A knowledge graph is constructed from these entity and relation triples which can be queried to obtain insights from the data. Furthermore, we integrate a suite of Large Language Models (LLMs) that can be used for QA and summarisation that is grounded in the included documents. KnowledgeHub is a unique tool that supports annotation, IE and QA, which gives the user full insight into the knowledge discovery pipeline.

Knowledge graphs (KGs) have transformed data management within the manufacturing industry, offering effective means for integrating disparate data sources through shared and structured conceptual schemas. However, harnessing the power of KGs can be daunting for non-experts, as it often requires formulating complex SPARQL queries to retrieve specific information. With the advent of Large Language Models (LLMs), there is a growing potential to automatically translate natural language queries into the SPARQL format, thus bridging the gap between user-friendly interfaces and the sophisticated architecture of KGs. The challenge remains in adequately informing LLMs about the relevant context and structure of domain-specific KGs, e.g., in manufacturing, to improve the accuracy of generated queries. In this paper, we evaluate multiple strategies that use LLMs as mediators to facilitate information retrieval from KGs. We focus on the manufacturing domain, particularly on the Bosch Line Information System KG and the I40 Core Information Model. In our evaluation, we compare various approaches for feeding relevant context from the KG to the LLM and analyze their proficiency in transforming real-world questions into SPARQL queries. Our findings show that LLMs can significantly improve their performance on generating correct and complete queries when provided only the adequate context of the KG schema. Such context-aware prompting techniques help LLMs to focus on the relevant parts of the ontology and reduce the risk of hallucination. We anticipate that the proposed techniques help LLMs to democratize access to complex data repositories and empower informed decision-making in manufacturing settings.

Recent research in the field of design engineering is primarily focusing on using AI technologies such as Large Language Models (LLMs) to assist early-stage design. The engineer or designer can use LLMs to explore, validate, and compare thousands of generated conceptual stimuli and make final choices. This was seen as a significant stride in advancing the status of the generative approach in computer-aided design. However, it is often difficult to instruct LLMs to obtain novel conceptual solutions and requirement-compliant in real design tasks, due to the lack of transparency and insufficient controllability of LLMs. This study presents an approach to leverage LLMs to infer Function–Behavior–Structure (FBS) ontology for high-quality design concepts. Prompting design based on the FBS model decomposes the design task into three sub-tasks including functional, behavioral, and structural reasoning. In each sub-task, prompting templates and specification signifiers are specified to guide the LLMs to generate concepts. User can determine the selected concepts by judging and evaluating the generated function–structure pairs. A comparative experiment has been conducted to evaluate the concept generation approach. According to the concept evaluation results, our approach achieves the highest scores in concept evaluation, and the generated concepts are more novel, useful, functional, and low cost compared to the baseline.

Large language models (LLMs) are increasingly deployed in information systems, including being used as second-stage rerankers in information retrieval pipelines, yet their susceptibility to recency bias has received little attention. We investigate whether LLMs implicitly favour newer documents by prepending artificial publication dates to passages in the TREC Deep Learning passage retrieval collections in 2021 (DL21) and 2022 (DL22). Across seven models, GPT-3.5-turbo, GPT-4o, GPT-4, LLaMA-3 8B/70B, and Qwen-2.5 7B/72B, ''fresh'' passages are consistently promoted, shifting the Top-10's mean publication year forward by up to 4.78 years and moving individual items by as many as 95 ranks in our listwise reranking experiments. Although larger models attenuate the effect, none eliminate it. We also observe that the preference of LLMs between two passages with an identical relevance level can be reversed by up to 25% on average after date injection in our pairwise preference experiments. These findings provide quantitative evidence of a pervasive recency bias in LLMs and highlight the importance of effective bias-mitigation strategies.

MotivationMetagenomic data increasingly reflect the coexistence of species from Archaea, Bacteria, Eukaryotes, and Viruses in complex environments. Taxonomic classification across the four superkingdoms is essential for understanding microbial communities, exploring genomic evolutionary relationships, and identifying novel species. This task is inherently imbalanced, uneven, and hierarchical. Genomic sequences provide crucial information for taxonomy classification, but many existing methods relying on sequence similarity to reference genomes often leave sequences misclassified due to incomplete or absent reference databases. Large language models offer a novel approach to extract intrinsic characteristics from sequences.ResultsWe present ICCTax, a classifier integrating the large language model HyenaDNA with complementary-view-based hierarchical metric learning and hierarchical-level compactness loss to identify taxonomic genomic sequences. ICCTax accurately classifies sequences to 155 genera and 43 phyla across the four superkingdoms, including unseen taxa. Across three datasets built with different strategies, ICCTax outperforms baseline methods, particularly on Out-of-Distribution data. On Simulated Marine Metagenomic Communities datasets from three oceanic sites, DairyDB-16S rRNA, Tara Oceans, and wastewater metagenomic datasets, it demonstrates strong performance, showcasing real-world applicability. ICCTax can further support identification of novel species and functional genes across diverse environments, enhancing understanding of microbial ecology.Availability and implementationCode is available at https://github.com/Ying-Lab/ICCTax.

Although Large language Model (LLM)-powered information extraction (IE) systems have shown impressive capabilities, current fine-tuning paradigms face two major limitations: high training costs and difficulties in aligning with LLM preferences. To address these issues, we propose a novel universal IE paradigm—the Self-Correcting Iterative Refinement (SCIR) framework—along with a Multi-task Bilingual (Chinese-English) Self-Correcting (MBSC) dataset containing over 100,000 entries. The SCIR framework achieves plug-and-play compatibility with existing LLMs and IE systems through its Dual-Path Self-Correcting module and feedback-driven optimization, thereby significantly reducing training costs. Concurrently, the MBSC dataset tackles the challenge of preference alignment by indirectly distilling GPT-4's capabilities into IE result detection models. Experimental results demonstrate that SCIR outperforms state-of-the-art IE methods across three key tasks— named entity recognition, relation extraction, and event extraction—achieving a 5.27 percent average improvement in span-based Micro-F1 while reducing training costs by 87 percent compared to baseline approaches. These advancements not only enhance the flexibility and accuracy of IE systems but also pave the way for lightweight and efficient IE paradigms.

The article considers the main aspects of evolution and performs a thorough analysis of the stages of formation of multimodal artificial intelligence systems (AIS). It was determined that in modern realities, artificial intelligence has undergone a transformational shift towards embracing multimodality in large language models. Modern approaches and ways of improving large language models by means of processing and generating a large amount of data are analyzed. The stages of transformation of artificial intelligence in the direction of multimodality of innovative development in large language models have been studied. The issue of verification and interaction of information systems with the surrounding world is considered. It was determined that they are inherently multimodal, multicomponent. Ways of improving large language models with the help of the ability to process and generate different data modalities are analyzed. It has been investigated that modern multimodal artificial intelligence systems are effectively used in various fields of science, education, and economics and require further development and improvement. It was determined that due to the rapid development of information technologies and systems in various spectrums of life, AI is experiencing a rapid modification, where generative models, which are becoming more and more perfect, deserve special attention. An overview of the architecture of the AnyGPT model is performed, where modalities are tokenized into discrete tokens, on the basis of which LLM performs multimodal perception and generation in autoregression. The methodology underlying AnyGPT was found to be multi-component, with the model demonstrating capabilities on par with specialized models in all assessment modalities tested. It has been established that tools designed to detect objects generated by artificial intelligence are in a state of development and are constantly being modified.

Purpose: This study examines the potential of large language models (LLMs) in the field of information systems auditing (ISA).Exploration of LLMs' internal knowledge is one of the most interesting research directions and concerns the range of information, knowledge such agents store without any access to external sources.The foundation of IS audit and control has evolved through the recognition of the need for strong information technology (IT) management by professional organizations, e.g., ISACA, business and government institutions.Other major areas for IS auditors are the issues of security and privacy of IT.The demand by the public and governments have generated concern for the protection of individual rights and business information, as well as the need to supervise the whole organization to ensure effective and efficient management.Design/methodology/approach: The research applied both quantitative and qualitative evaluation methods using 146 official CISA certification questions to assess the auditing knowledge of seven large language models.The analysis measured model accuracy, inter-agent agreement, and A-bias tendencies, while qualitative examination identified reasoning errors and interpretation patterns in incorrectly answered questions.Findings: The study found that large language models achieved an average accuracy of 81,8% on CISA certification questions, with Claude Sonnet 4 and Chat GPT-4.1 performing best at 88.4%.Some models showed certain reasoning limitations, such as bias toward specific answer options. Research limitations/implications:The main limitation of this study is that the evaluation was based solely on multiple-choice CISA certification questions, which may not fully capture the complexity of real-world auditing scenarios.Practical implications: The study indicates that large language models can enhance the efficiency of audit preparation and training by automating knowledge-intensive tasks.Originality/value: The paper highlights the growing role of generative artificial intelligence in professional certification and auditing practice.

The increasing volume of regulatory documentation in construction and engineering complicates decision-making and raises the risk of errors and non-compliance. Existing search systems fail to provide sufficient semantic understanding of user needs, while large language models suffer from hallucinations and incomplete answers. This paper presents a hybrid intelligent assistant that integrates retrieval-augmented generation, ontological modeling, and multi-agent architecture to address these limitations. The system extracts entities and situations from regulatory documents to build an ontological domain model, enabling both semantic interpretation and automated consistency verification of generated responses. Vector similarity measures are combined with concept-level ontological relationships to improve retrieval accuracy. A multi-agent architecture ensures modularity and scalability, supporting document analysis, query history tracking, personalized response generation based on user context, and precise reference linking to primary sources. The proposed approach is evaluated in the construction domain, where practitioners must simultaneously consider standards at multiple regulatory levels. The practical contribution lies in reducing compliance costs and risks while increasing efficiency in handling complex multi-level regulatory frameworks. Future research will focus on cross-domain implementation and comprehensive evaluation of system performance.

The challenges inherent in developing, implementing, and supporting information systems in organizations with global operations have led to an increased focus on the incorporation of global issues in information systems curricula. This paper describes a means of integrating global issues in the introductory MIS course. The approach employs a spreadsheet project that requires students to design, develop, and document a Microsoft Excel workbook to consolidate global sales data for a business organization with subsidiaries in seven countries. After completing the assignment, students use a cooperative learning technique to examine global issues in information systems.

GeoPredict-LLM: Intelligent tunnel advanced geological prediction by reprogramming large language models