Data mining for building knowledge bases: techniques, architectures and applications

The availability of massive sets of unstructured data has opened up new opportunities for businesses to gain meaningful insights into what is currently happening in their organization and market place. Unfortunately, unstructured data is often challenging to work with, as it requires specialized toolsets, techniques, knowledge and skills to engage with the data. This study used a systematic review (SLR) process to explore the current and conversant state of affairs in the field of text mining (TM) techniques that exist for the processing of natural language in unstructured text datasets across a broad expanse of applications and domains. A comprehensive literature search yielded 1022 eligible articles from five prominent bibliographic databases, narrowed down to 89 articles for review. Information related to TM techniques, the TM process, applications, challenges and recommendations for the improvement of TM results were extracted and synthesized. Eighteen TM techniques were identified and used to complete a variety of tasks such as data pre-processing, information retrieval, information extraction, text classification, clustering, topic modeling, text summarization and sentiment analysis or opinion mining. No single TM technique may be suitable for all text representation and extraction requirements. Therefore, context and applicability are key factors for TM technique selection.KeywordsText miningText mining techniquesUnstructured dataKnowledge discovery

Biomedical information is available to research and development scientists as unstructured text in the form of scientific manuscripts and reports published in the literature and elsewhere. Scientists focused on specific research programs are burdened with surveying vast numbers of publications and reports to acquire information relevant to their efforts. Employing technology as a research aid provides a mechanism to cope with information overload that characterizes the R&D environment. Text mining can extract knowledge from large corpora of biomedical text and make it available to support scientific research and knowledge collections [1, 2] and intelligent PDF reader tools able to search content and find related articles [3] are available; however, such reader tools are typically desktop applications limited to specific platforms and data sources so they cannot easily support broad based integrated scientific search needs for a dispersed R&D organization with a wide variety of content needs. Our team has developed a web-browser based document reader with a built-in exploration tool and automatic concept extraction from biomedical text content. This provides R&D scientists with a simple tool to aid finding, reading, and exploring documents relevant to focused research objectives. The tool, Shangri-Docs, combines a document reader with automatic concept extraction and highlighting of relevant terms based on carefully selected ontologies combined with our custom corporate enterprise taxonomy. Shangri-Docs provides the ability to evaluate a wide variety of document formats (e.g. PDF, Word, PPT, text, etc.) and exploits the linked nature of the Web and personal content by performing searches on content from public sites (e.g. Wikipedia, PubMed) and privately cataloged databases simultaneously. Shangri-Docs incorporates Apache cTAKES (clinical Text Analysis and Knowledge Extraction System) [4] and Unified Medical Language System (UMLS) to automatically identify and highlight terms and concepts, such as specific pathology, disease, drug, and biological terms mentioned in the text. cTAKES was originally designed specifically to extract information from clinical medical records. We have extended cTAKES automatic knowledge extraction process to include the R&D biomedical research domain by improving the ontology guided information extraction process. Shangri-Docs could be adapted to other science fields and further customized across our R&D scientific community via our open source, cloud-based, data management system. [1] Funk, et.al., BMC Bioinformatics 2014, 15:59 doi:10.1186/1471-2105-15-59 [2] Kang et al., BMC Bioinformatics 2014, 15:64 doi:10.1186/1471-2105-15-64 [3] Utopia Documents, http://utopiadocs.com [4] Apache cTAKES, http://ctakes.apache.org Citation Format: Chris Mattmann, Lauren Intagliata, Selina Chu, Garth McGrath, Giuseppe Totaro, Daniel Civello, David Ballard, Jeffrey Long, Nipurn Doshi, Shivika Thapar, Michael Livstone, Paul Ramirez, Maureen Cronin. Shangri-Docs: a browser based tool for document exploration and automatic knowledge extraction from unstructured biomedical text. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 5283.

This paper investigates the impact of knowledge extraction methods on the effectiveness of RAG (Retrieval-Augmented Generation) systems that utilize knowledge graphs. It highlights that the quality of the knowledge graph, which is formed using various knowledge extraction methods, is crucial for overcoming limitations of large language models (LLMs), such as “hallucinations”. The paper analyzes the architectures of LightRAG and GraphRAG, emphasizing that the selection of an optimal knowledge extraction strategy depends on specific tasks and the subject area. LLMs have advanced significantly, but they have limitations, including generating factually incorrect information (“hallucinations”) and possessing “outdated knowledge”. RAG systems were proposed to address these issues by combining LLMs with external knowledge bases. This approach reduces hallucinations, ensures factual accuracy, solves the problem of outdated knowledge, and increases transparency. Knowledge graphs are powerful tools for structuring information, consisting of entities (nodes) and relations (edges). They enhance RAG systems by enabling more precise and contextually grounded retrieval compared to keyword-based searches. The quality of a knowledge graph depends on the knowledge extraction methods used, which include named entity recognition (NER), relation extraction (RE), entity linking (EL), and event extraction. Different methods, such as rule-based, classical machine learning, and deep learning approaches, have varying trade-offs in terms of accuracy, completeness, and scalability. Entity linking and knowledge graph completion are also crucial for accuracy and richness. LightRAG and GraphRAG are two main graph-based RAG systems. LightRAG uses the knowledge graph as a quick reference, requiring high precision in knowledge extraction to avoid noise degradation. GraphRAG uses the knowledge graph as a domain model, where completeness of extracted knowledge is more critical, though systematic errors are still harmful. Both systems rely on LLMs for knowledge extraction, which makes them dependent on the LLM’s quality and the size of document fragments processed. The theoretical analysis confirms that the effectiveness of RAG systems is critically dependent on knowledge extraction methods. The quality, completeness, and accuracy of the knowledge graph directly influence the RAG system’s ability to provide relevant, accurate, and truthful answers. Different RAG system architectures like LightRAG and GraphRAG have distinct requirements for knowledge graph characteristics, prioritizing either accuracy or completeness in knowledge extraction.

Making use of knowledge bases to answer questions (KBQA) is a key direction in question answering systems. Researchers have developed a diverse range of methods to address this problem, but there are still some limitations with the existing methods. Specifically, the existing neural network-based methods for KBQA have not taken advantage of the recent "matching-aggregation" framework for the sequence matching, and when representing a candidate answer entity, they may not choose the most useful context of the candidate for matching. In this paper, we explore the use of a "matching-aggregation" framework to match candidate answers with questions. We further make use of question-specific contextual relations to enhance the representations of candidate answer entities. Our complete method is able to achieve state-of-the-art performance on two benchmark datasets: WebQuestions and SimpleQuestions.

Machine-readable semantic knowledge in the form of taxonomies (i.e., a collection of is-a edges) has proved to be beneficial in an array of NLP tasks including inference, textual entailment, question answering and information extraction. Such widespread utility of taxonomies has led to multiple large-scale manual efforts towards taxonomy induction such as WordNet and Cyc. However, manual construction of taxonomies is time-intensive, and usually, requires substantial annotation efforts by domain experts. Furthermore, the resulting taxonomies suffer from low coverage and are unavailable for specific domains or languages. Therefore, in recent years, there has been a growing body of work, which aims to induce taxonomies automatically, either from semi-structured knowledge resources or unstructured text. In this thesis, we focus on the task of automated taxonomy induction under a variety of different settings. We first focus on taxonomy induction from the largest semi-structured knowledge resource, i.e., Wikipedia. More specifically, we introduce a set of novel heuristics aimed towards inducing a large-scale taxonomy from the English Wikipedia categories network. We also propose a novel comprehensive path-based evaluation framework for taxonomies. Taxonomy induced using our approach significantly outperforms the state of the art across both edge-based as well as path-based evaluation metrics. Moreover, our experiments also demonstrate that high accuracy in edge-based evaluation metrics does not always translate to high-accuracy in path-based evaluation metrics. Subsequently, we propose a novel approach, which leverages the interlanguage links of Wikipedia to induce taxonomies in other Wikipedia languages. Compared to the state of the art, our approach is simpler, more principled, and results in taxonomies that are significantly more accurate across both edge-based and path-based evaluation metrics. In the second part of this thesis, we focus on taxonomy induction from unstructured text. We propose a novel approach towards taxonomy induction from an input vocabulary of seed terms. Unlike previous approaches, which extract singular hypernym edges for terms, our approach utilizes a novel probabilistic framework to extract long-range hypernym subsequences. Taxonomy induction from the extracted subsequences is cast as an instance of the minimum-cost flow optimization problem on a carefully-designed flow network. Through experiments, we demonstrate that our approach outperforms the state of the art across four languages. We also show that our approach is robust to the presence of noise in the input vocabulary. Our approach facilitates the relaxation of many simplifying assumptions, which were employed by previous taxonomy induction approaches. As a result, our work serves to automate the process of taxonomy induction from unstructured text in the true sense. Finally, we introduce a task of discovering and generalizing lexicalized templates from the titles of Wikipedia entities. The experimental results on this task demonstrate that taxonomies, which perform better on our proposed path-based metrics, result in a more accurate set of generalizations for a given set of entities. In summary, this thesis proposes new approaches towards automated taxonomy induction. It improves upon the state of the art in a variety of different settings. It also relaxes many simplifying assumptions that limited the applicability of prior approaches.

Nowadays, most of the data on the Web is still in the form of unstructured text. Knowledge extraction from unstructured text is highly desirable but extremely challenging due to the inherent ambiguity of natural language. In this article, we present an architecture of an information extraction system based on the concept of Embedded Controlled Language that allows for extracting formal semantic knowledge from an unstructured text corpus. Moreover, the presented approach has a potential to support multilingual input and output.

Link prediction on knowledge graphs is useful in numerous application areas such as semantic search, question answering, entity disambiguation, enterprise decision support, recommender systems and so on. While many of these applications require a reasonably quick response and may operate on data that is constantly changing, existing methods often lack speed and adaptability to cope with these requirements. This is aggravated by the fact that knowledge graphs are often extremely large and may easily contain millions of entities rendering many of these methods impractical. In this paper, we address the weaknesses of current methods by proposing Random Semantic Tensor Ensemble (RSTE), a scalable ensemble-enabled framework based on tensor factorization. Our proposed approach samples a knowledge graph tensor in its graph representation and performs link prediction via ensembles of tensor factorization. Our experiments on both publicly available datasets and real world enterprise/sales knowledge bases have shown that our approach is not only highly scalable, parallelizable and memory efficient, but also able to increase the prediction accuracy significantly across all datasets.

Creating entity network graphs is a manual, time consuming process for an intelligence analyst. Beyond the traditional big data problems of information overload, individuals are often referred to by multiple names and shifting titles as they advance in their organizations over time which quickly makes simple string or phonetic alignment methods for entities insufficient. Conversely, automated methods for relationship extraction and entity disambiguation typically produce questionable results with no way for users to vet results, correct mistakes or influence the algorithm's future results. We present an entity disambiguation tool, DRADIS, which aims to bridge the gap between human-centric and machinecentric methods. DRADIS automatically extracts entities from multi-source datasets and models them as a complex set of attributes and relationships. Entities are disambiguated across the corpus using a hierarchical model executed in Spark allowing it to scale to operational sized data. Resolution results are presented to the analyst complete with sourcing information for each mention and relationship allowing analysts to quickly vet the correctness of results as well as correct mistakes. Corrected results are used by the system to refine the underlying model allowing analysts to optimize the general model to better deal with their operational data. Providing analysts with the ability to validate and correct the model to produce a system they can trust enables them to better focus their time on producing higher quality analysis products.

The most challenging task in any modern reasoning system is that it has been completely relied on automatic knowledge acquisition from the unstructured text and filtering out the structured information from it has turned out to be the most crucial task of Information Retrieval systems. In this paper, we have proposed a system that can recognize the potential named entities from the Twitter streams and link them to the appropriate real world knowledge entities. Besides, it has performed many semantic functions such as entity disambiguation, contextual similarity, type induction, and semantic labeling, to augment the semantic score of the entity and provide the rich entity feature space to quantitatively enhance entity retrieval accuracy. Nevertheless, we have leveraged a model to alleviate the entity imbalance present over the collected Twitter Streams and effectively utilized the contextual relatedness between the candidate entity sets. Eventually, we have proposed a probabilistic approach to deal with topic modeling and effectively disambiguate the entities by clustering the entities into its appropriate entity domain. The proposed Latent Dirichlet Allocation (LDA) model has been categorically distinguished the topics for clustering between the candidate entities and fix the exact true mentions occurred in the Knowledge Base such as DBpedia. We have also demonstrated the performance and accuracy rate of the proposed system and evaluated the results with the collected Twitter Streams for the month of August, 2016. The empirical results have shown that it has outperformed the existing state-of-the-art systems and proved that the proposed system given here has gradual accuracy rate against the conventional systems.

Nowadays, the broadcasting of news via social media networks is almost provided in a textual format. The nature of the broadcasted text is considered as unstructured text. Text mining techniques play an essential role in converting the unstructured text into informative knowledge. It has been observed that there is no research has addressed the textual analysis of Arabic newspapers on social media. Accordingly, this paper attempts to bridge this gap through building on related studies and applying various text mining techniques on a new under-researched context. 62,327 posts were collected from 24 Arab Gulf newspapers pages on Facebook. Results indicated that most of the discussed issues in the Arab Gulf region newspapers are related to trade, petroleum, and development. In addition, results revealed that the United Arab of Emirates (UAE) newspapers represent the source that is highly discussing issues regarding trade and economy followed by the Kingdom of Saudi Arabia (KSA) newspapers. Furthermore, results indicated that the financial and health-care issues news in the Arab Gulf region were highly tackled by Alkhaleej (UAE) newspaper. Besides, results pointed out that KSA newspapers, is on the top in disseminating issues regarding education.

Domain-specific knowledge is often recorded by experts in the form of unstructured text. For example, in the medical domain, clinical notes from electronic health records contain a wealth of information. Similar practices are found in other domains. The challenge we discuss in this paper is how to identify and extract part names from technicians repair notes, a noisy unstructured text data source from General Motors’ archives of solved vehicle repair problems, with the goal to develop a robust and dynamic reasoning system to be used as a repair adviser by service technicians. In the present work, we discuss two approaches to this problem. We present an algorithm for ontology-guided entity disambiguation that uses existing knowledge sources, such as domain-specific taxonomies and other structured data. We illustrate its use in the automotive domain, using GM parts ontology and the unit structure of repair manuals text to build context models, which are then used to disambiguate mentions of part-related entities in the text. We also describe extraction of part names with a small amount of annotated data using hidden Markov models (HMM) with shrinkage, achieving an f-score of approximately 80%. Next, we used linear-chain conditional random fields (CRF) in order to model observation dependencies present in the repair notes. Using CRF did not lead to improved performance, but a slight improvement over the HMM results was obtained by using a weighted combination of the HMM and CRF models.

It is an emerging research topic in NLP to joint extraction of knowledge including entities and relations from unstructured text and representing them as meaningful triplets. Despite significant progresses made by recent deep neural network based solutions, these methods still confront the overlapping issue that different relational triplets may have overlapped entities in a sentence, and it is troublesome to address this issue by current solutions. In this paper, we propose a novel multiple tagging model to address the overlapping issue and extract knowledge from unstructured text. Specifically, we devise a multiple tagging scheme that transforms the problem of joint entity and relation extraction into a multiple sequence tagging problem. By using GRU as the building block for encoding-decoding, the proposed model is capable of handling the triplet overlapping problem because the decoder layer allows one entity to take part in more than one triplet. The whole network is end-to-end trianable and outputs all triplets in a sentence directly. Experimental results on the NYT and KBP benchmarks demonstrate that the proposed model siginificantly improves the recall of triplet, and consequently, achieving the new state-of-the-art in the task of triplet extraction on both datasets.

Named entity recognition (NER) systems are important for extracting useful information from unstructured data sources. It is known that large domain dictionaries help in improving extraction performance of NER. Unstructured text usually contains entity mentions that are different from their standard dictionary form. Approximate matching is important to identify the correct dictionary entity for such variants. This is a challenging problem, as every entity in the dictionary is a candidate match for the variant. In this paper, we propose a novel approach for efficient approximate dictionary matching. The key idea is to compare a given query only against a set of most likely candidate matches from the dictionary so as to achieve substantial reduction in the number of matching operations. In order to enable this, the proposed approach first performs clustering of similar entities and then represents each cluster with a profile matrix, which stores the probability of an occurrence of a particular character at a specific location in the entity string. Thus, the dictionary is represented with a set of profile matrices, which are much smaller than the actual number of entities. A given query entity is first matched against the profiles and the clusters corresponding to top-K best scoring profiles are selected to obtain a list of most likely matching candidates. The query is then compared with each candidate match entity and the approximate match is declared if both the query and the candidate entity are within acceptable edit distance threshold. We have performed rigorous evaluation of our approach on several publicly available datasets. The proposed algorithm outperforms alternative approaches in detecting approximately matching entities for a given query using far lesser number of comparison operations.

Relation extraction is an important task for understanding text. In the big data era, automatic relation extraction from unstructured texts is urgently needed for structured information organization and information analysis. In this paper, we survey the automatic relation extraction methods, especially the traditional machine learning on closed data set and open information environment such as Web, including supervised and semi-supervised methods. And then, we discuss the applications based on relation extraction such as event extraction and QA systems.

Government hotline plays a significant role in meeting the demands of the people and resolving social conflicts in China. In this paper, we propose an automatic work‐order assignment method based on event extraction and external knowledge to address the problem of low efficiency with manual assignment for Chinese government hotline. Our proposed assignment method is composed of four parts: (1) Semantic encoding layer, which extracts semantic information from the work‐order text and obtains semantic representation vectors with contextual feature information. (2) Event extraction layer which extracts the local features and global features from the semantic representation vectors with the help of the CRF network to enhance event extraction effect. (3) External knowledge embedding layer, which integrates ‘rights and responsibilities lists’ with the historical information of the work‐order to assist assignment. (4) Assignment layer which completes work‐order assignment by combining two output vectors from event extraction layer and external knowledge embedding layer. Experimental results show our proposed method can achieve better assignment performance compared with several baseline methods.