Retrieval Augmented Generation (RAG) for Fintech: Agentic Design and Evaluation

This research addresses the problem of automatic keyphrase extraction from large documents and back of the book indexing. The potential benefits of automating this process are far reaching, from improving information retrieval in digital libraries, to saving countless man-hours by helping professional indexers creating back of the book indexes. The dissertation introduces a new methodology to evaluate automated systems, which allows for a detailed, comparative analysis of several techniques for keyphrase extraction. We introduce and evaluate both supervised and unsupervised techniques, designed to balance the resource requirements of an automated system and the best achievable performance. Additionally, a number of novel features are proposed, including a statistical informativeness measure based on chi statistics; an encyclopedic feature that taps into the vast knowledge base of Wikipedia to establish the likelihood of a phrase referring to an informative concept; and a linguistic feature based on sophisticated semantic analysis of the text using current theories of discourse comprehension. The resulting keyphrase extraction system is shown to outperform the current state of the art in supervised keyphrase extraction by a large margin. Moreover, a fully automated back of the book indexing system based on the keyphrase extraction system was shown to lead to back of the book indexes closely resembling those created by human experts.

This paper proposes a non-domain-specific metadata ontology as a core component in a semantic model-based document management system (DMS), a potential contender towards the enterprise information systems of the next generation. What we developed is the core semantic component of an ontology-driven DMS, providing a robust semantic base for describing documents' metadata. We also enabled semantic services such as automated semantic translation of metadata from one domain to another. The core semantic base consists of three semantic layers, each one serving a different view of documents' metadata. The core semantic component's base layer represents a non-domain-specific metadata ontology founded on ebRIM specification. The main purpose of this ontology is to serve as a meta-metadata ontology for other domain-specific metadata ontologies. The base semantic layer provides a generic metadata view. For the sake of enabling domain-specific views of documents' metadata, we implemented two domain-specific metadata ontologies, semantically layered on top of ebRIM, serving domain-specific views of the metadata. In order to enable semantic translation of metadata from one domain to another, we established model-to-model mappings between these semantic layers by introducing SWRL rules. Having the semantic translation of metadata automated not only allows for effortless switching between different metadata views, but also opens the door for automating the process of documents long-term archiving. For the case study, we chose judicial domain as a promising ground for improving the efficiency of the judiciary by introducing the semantics in this field.

Retrieval-Augmented Generation (RAG) systems have shown significant potential for domain-specific Question Answering (QA) tasks, although persistent challenges in retrieval precision and context selection continue to hinder their effectiveness. This study introduces Document Graph RAG (GraphRAG), a novel framework that bolsters retrieval robustness and enhances answer generation by incorporating Knowledge Graphs (KGs) built upon a document’s intrinsic structure into the RAG pipeline. Through the application of the Design Science Research methodology, we systematically design, implement, and evaluate GraphRAG, leveraging graph-based document structuring and a keyword-based semantic linking mechanism to improve retrieval quality. The evaluation, conducted on well-established datasets including SQuAD, HotpotQA, and a newly developed manufacturing dataset, demonstrates consistent performance gains over a naive RAG baseline across both retrieval and generation metrics. The results indicate that GraphRAG improves Context Relevance metrics, with task-dependent optimizations for chunk size, keyword density, and top-k retrieval further enhancing performance. Notably, multi-hop questions benefit most from GraphRAG’s structured retrieval strategy, highlighting its advantages in complex reasoning tasks.

Entity disambiguation is the task of mapping ambiguous terms in natural-language text to its entities in a knowledge base. Most disambiguation systems focus on general purpose knowledge bases like DBpedia but leave out the question how those results generalize to more specialized domains. This is very important in the context of Linked Open Data, which forms an enormous resource for disambiguation. We implement a ranking-based Learning To Rank disambiguation system and provide a systematic evaluation of biomedical entity disambiguation with respect to three crucial and well-known properties of specialized disambiguation systems. These are i entity context, i.e. the way entities are described, ii user data, i.e. quantity and quality of externally disambiguated entities, and iii quantity and heterogeneity of entities to disambiguate, i.e. the number and size of different domains in a knowledge base. Our results show that i the choice of entity context that is used to attain the best disambiguation results strongly depends on the amount of available user data, ii disambiguation results with large-scale and heterogeneous knowledge bases strongly depend on the entity context, iii disambiguation results are robust against a moderate amount of noise in user data and iv some results can be significantly improved with a federated disambiguation approach that uses different entity contexts. Our results indicate that disambiguation systems must be carefully adapted when expanding their knowledge bases with special domain entities.

Learning to rank can be applied to a wide variety of applications in information retrieval and natural language processing. Typical applications are document retrieval, expert search, definition search, meta-search, personalized search, online advertisement, collaborative filtering, question answering, keyphrase extraction, document summarization, and machine translation.

Factoid question answering (QA) has recently benefited from the development of deep learning (DL) systems. Neural network models outperform traditional approaches in domains where large datasets exist, such as SQuAD (ca. 100,000 questions) for Wikipedia articles. However, these systems have not yet been applied to QA in more specific domains, such as biomedicine, because datasets are generally too small to train a DL system from scratch. For example, the BioASQ dataset for biomedical QA comprises less then 900 factoid (single answer) and list (multiple answers) QA instances. In this work, we adapt a neural QA system trained on a large open-domain dataset (SQuAD, source) to a biomedical dataset (BioASQ, target) by employing various transfer learning techniques. Our network architecture is based on a state-of-the-art QA system, extended with biomedical word embeddings and a novel mechanism to answer list questions. In contrast to existing biomedical QA systems, our system does not rely on domain-specific ontologies, parsers or entity taggers, which are expensive to create. Despite this fact, our systems achieve state-of-the-art results on factoid questions and competitive results on list questions.

Background: Web documents display information in the form of natural language text which is not understandable by machines. To search specific information from sea of web documents has become very challenging as it shows many unwanted non relevant documents along with relevant documents. To retrieve relevant information semantic knowledge can be stored in the domain specific ontology which helps in understanding user’s need to retrieve relevant information. Methods: In this paper, framework for extracting and visualising semantic knowledge has been designed. Proposed approach is based on the assumption that semantics of text can be extracted by creating syntactic structure of the text. To extracts syntactic structure Stanford parser has been used. Parsing of corpus text is done to obtain morphological structures which is in more machine readable format, and thus provides a better structure for constructing syntactic-semantic rules manually. The tagged form of each sentence is taken and set of rules based on dependency relationship are build manually. Sentence level analysis is performed for concepts generation, for properties and for hierarchical relation extraction using dependency parse tree as a means for relation extraction. Results: Extracted concepts and relation among various entities constitute knowledge base in the form of ontology. Conclusion: Proposed information extraction model successfully filter the desired information from the large ocean of internet and create semantic structure to represent data in standard machine understandable format which explain the details about entities along with their properties and their relationship.

Representation Learning and Learning from Limited Labeled Data for Community Question Answering

The keyphrase extraction task is a fundamental and challenging task designed to extract a set of keyphrases from textual documents. Keyphrases are essential to assist publishers in indexing documents and readers in identifying the most relevant ones. They are short phrases composed of one or more terms used to represent a textual document and its main topics. In this article, we extend our research on C-Rank, which is an unsupervised approach that automatically extracts keyphrases from single documents. C-Rank uses concept-linking to link concepts in common between single documents and an external background knowledge base. We advance our study over C-Rank by evaluating it using different concept-linking approaches - Babelfy and DBPedia Spotlight. We evaluated C-Rank on data sets composed of academic articles, academic abstracts, and news articles. Our findings indicate that C-Rank achieves state-of-the-art results extracting keyphrases from scientific documents by experimentally comparing it to existing unsupervised approaches.

"The purpose of the present paper is to research a rule-based approach for transforming X3D (eXtensible 3D) models to RDF (Resource Description Framework). The transformation is performed by using the RDF Mapping Language (RML). Its advantages are summarized, which are mainly due to the fact that the rules created build a knowledge base. By applying SPARQL (SPARQL Protocol and RDF Query Language) queries to it, the possibility of explore in order to validate and improve the defined RML rules themselves, is pointed out. An approach for reversing from the RDF triples to the original X3D in a unique way is considered, and the types of SPARQL queries needed for its implementation are systematized. Rules are formulated for all elements defined in the X3D schema, their attributes and properties are described. Their accessibility is ensured. The conversion of X3D models to RDF is confirmed to be consistent with guidelines and best practices for creating accessible, understandable, and reusable ontologies on the Web. The systematized SPARQL query types for reversing from RDF triples to the original X3D are checked for specific elements and sample data, and the obtained results establish their correctness. The prerequisites and limitations of the represented approach are explained. The proposed approach allows building a comprehensive knowledge base that includes the RML rules, the transformed X3D models and the domain-specific ontology and its use to analyzing data and semantic reasoning. The electronic libraries that include 3D content could take advantage from the benefits and possible future applications of the solutions discussed in this study."

Inherent heterogeneity and distribution of knowledge strongly prevent knowledge from sharing and reusing among different agents and software entities, and a formal ontology has been viewed as a promising means to tackle this problem. In this paper, a domain-specific formal ontology of archaeology is presented. The ontology mainly consists of three parts: archaeological categories, their relationships and axioms. The ontology not only captures the semantics of archaeological knowledge, but also provides archaeology with an explicit and formal specification of a shared conceptualization, thus making archaeological knowledge shareable and reusable across humans and machines in a structured fashion. Further, we propose a method to verify onlology correctness based on the individuals of categories. As applications of the ontology, we have developed an ontology-driven approach to knowledge acquisition from archaeological text and a question answering system for archaeological knowledge.

In this paper, we propose the construction of a web-based Question Answering (QA) system for restricted domain, which combines three resource information databases for the retrieval mechanism, including a Question&Answer database, a special domain documents database and the web resource retrieved by Google search engine. We describe a new retrieval technique of integrating a probabilistic technique based on OkapiBM25 and a semantic analysis which based on the ontology of HowNet knowledge base and a special domain HowNet created for the restricted domain. Furthermore, we provide a method of question expansion by computing word semantic similarity. The system is first developed for a middle-size domain of sightseeing information. The experiments proved the efficiency of our method for restricted domain and it is feasible to transfer to other domains expediently using the proposed method.

In this article we collect a corpus of texts which operate with a controlled language (ASD Simplified Technical English) in order to facilitate the development of a new domain-specific ontology (the aircraft structure) based on a technical discipline (aeronautical engineering) included in the so called “hard” sciences. This new repository should be compatible with the Core Ontology and the corresponding English Lexicon in FunGramKB (a multipurpose lexico-conceptual knowledge base for natural language processing (NLP)), and, in the same vein, should eventually give support to aircraft maintenance management systems. By contrast, in previous approaches we applied a stepwise methodology for the construction of a domain-specific subontology compatible with FunGramKB systems in criminal law, but the high occurrence of terminological banalisation and the scarce number of specific terms, due to the social nature of the discipline, were added problems to the most common NLP difficulties (polysemy and ambiguity). Taking into consideration previous results and the complexity of this task, here we only intend to take the first step towards the modelling of the aircraft ontology: the development of its taxonomic hierarchy. Consequently, the hierarchy starts with the whole system (i.e., an aircraft) and follows the traditional decomposition of the system down to the elementary components (top-down approach). At the same time, we have collected a corpus of 2,480 files of aircraft maintenance instructions, courtesy of Airbus in Seville. For the bottom-up approach (under construction), we consult specialised references end explore the corpus through the identification and extraction of term candidates with DEXTER, an online multilingual workbench especially designed for the discovery and extraction of terms.

Developer communities built around software products, like the SAP Community Network, provide a knowledge base for reocurring problems and their solutions. Due to the large amount of content maintained in such communities, e.g., in forums, finding relevant solutions is a major challenge beyond the scope of common keyword-based search engines. In fact, it is measured that around 50% of the forum questions of our particular scenario have already been answered at the time they are posted. We target this challenge by an entity aware search, which exploits structured knowledge, such as domain-specific ontologies, for both query interpretation and creation of document indexes. The system takes a natural language query as input, interprets it as an entity graph, matches this graph with pre-processed content and supports the user in refining his query based on the top-k relevant entities. Results are presented in a user interface that supports faceted search based on entities. Additionally, the user interface is structured according to possible search intentions of users. The evaluation of our system on the SCN scenario yields that the top 5 entities in user queries are recognized with a precision of 83% compared to 61% of state of the art algorithms.

Natural Language Processing (NLP) has been transformed by the advent of Large Language Models (LLMs), yet they continue to face challenges such as factual inaccuracies and "hallucinations." Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm to mitigate these issues by grounding LLMs in external knowledge. However, standard or "Naive" RAG implementations often suffer from limitations in retrieval precision and contextual coherence. This paper presents a systematic analysis of RAG systems by designing, implementing, and empirically comparing two primary frameworks: a baseline Naive RAG and an enhanced Advanced RAG. Our Advanced RAG incorporates sophisticated pre-retrieval (query rewriting) and post-retrieval (reranking) stages to enhance performance. To evaluate these frameworks in a practical, domain-specific context, we constructed a novel knowledge base from the Georgia Institute of Technology’s public web data and benchmarked them using the RAGAS framework for a multi-faceted evaluation. Our results demonstrate that the Advanced RAG model significantly outperforms the Naive RAG, achieving an 18.07% improvement in Answer Correctness and an 11.43% increase in Context Recall. This study underscores the critical role of refined retrieval and reranking pipelines in developing robust, accurate, and context-aware RAG systems for real-world applications.