Extracting Semantic Knowledge from Unstructured Text Using Embedded Controlled Language

The majority of data on computers nowadays is in the form of unstructured data and unstructured text. The inherent ambiguity of natural language makes it incredibly difficult but also highly profitable to find hidden information or comprehend complex semantics in unstructured text. In this paper, we present the combination of natural language processing (NLP) and convolution neural network (CNN) hybrid architecture called automated analysis of unstructured text using machine learning (AAUT-ML) for the detection of complex semantics from unstructured data that enables different users to make understand formal semantic knowledge to be extracted from an unstructured text corpus. The AAUT-ML has been evaluated using three datasets data mining (DM), operating system (OS), and data base (DB), and compared with the existing models, i.e., YAKE, term frequency-inverse document frequency (TF-IDF) and text-R. The results show better outcomes in terms of precision, recall, and macro-averaged F1-score. This work presents a novel method for identifying complex semantics using unstructured data.

Biomedical information is available to research and development scientists as unstructured text in the form of scientific manuscripts and reports published in the literature and elsewhere. Scientists focused on specific research programs are burdened with surveying vast numbers of publications and reports to acquire information relevant to their efforts. Employing technology as a research aid provides a mechanism to cope with information overload that characterizes the R&D environment. Text mining can extract knowledge from large corpora of biomedical text and make it available to support scientific research and knowledge collections [1, 2] and intelligent PDF reader tools able to search content and find related articles [3] are available; however, such reader tools are typically desktop applications limited to specific platforms and data sources so they cannot easily support broad based integrated scientific search needs for a dispersed R&D organization with a wide variety of content needs. Our team has developed a web-browser based document reader with a built-in exploration tool and automatic concept extraction from biomedical text content. This provides R&D scientists with a simple tool to aid finding, reading, and exploring documents relevant to focused research objectives. The tool, Shangri-Docs, combines a document reader with automatic concept extraction and highlighting of relevant terms based on carefully selected ontologies combined with our custom corporate enterprise taxonomy. Shangri-Docs provides the ability to evaluate a wide variety of document formats (e.g. PDF, Word, PPT, text, etc.) and exploits the linked nature of the Web and personal content by performing searches on content from public sites (e.g. Wikipedia, PubMed) and privately cataloged databases simultaneously. Shangri-Docs incorporates Apache cTAKES (clinical Text Analysis and Knowledge Extraction System) [4] and Unified Medical Language System (UMLS) to automatically identify and highlight terms and concepts, such as specific pathology, disease, drug, and biological terms mentioned in the text. cTAKES was originally designed specifically to extract information from clinical medical records. We have extended cTAKES automatic knowledge extraction process to include the R&D biomedical research domain by improving the ontology guided information extraction process. Shangri-Docs could be adapted to other science fields and further customized across our R&D scientific community via our open source, cloud-based, data management system. [1] Funk, et.al., BMC Bioinformatics 2014, 15:59 doi:10.1186/1471-2105-15-59 [2] Kang et al., BMC Bioinformatics 2014, 15:64 doi:10.1186/1471-2105-15-64 [3] Utopia Documents, http://utopiadocs.com [4] Apache cTAKES, http://ctakes.apache.org Citation Format: Chris Mattmann, Lauren Intagliata, Selina Chu, Garth McGrath, Giuseppe Totaro, Daniel Civello, David Ballard, Jeffrey Long, Nipurn Doshi, Shivika Thapar, Michael Livstone, Paul Ramirez, Maureen Cronin. Shangri-Docs: a browser based tool for document exploration and automatic knowledge extraction from unstructured biomedical text. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 5283.

A large amount of structured information is buried in unstructured text. Information extraction systems can extract structured relations from the documents and enable sophisticated, SQL-like queries over unstructured text. Information extraction systems are not perfect and their output has imperfect precision and recall (i.e., contains spurious tuples and misses good tuples). Typically, an extraction system has a set of parameters that can be used as “knobs” to tune the system to be either precision- or recall-oriented. Furthermore, the choice of documents processed by the extraction system also affects the quality of the extracted relation. So far, estimating the output quality of an information extraction task has been an ad hoc procedure, based mainly on heuristics. In this article, we show how to use Receiver Operating Characteristic (ROC) curves to estimate the extraction quality in a statistically robust way and show how to use ROC analysis to select the extraction parameters in a principled manner. Furthermore, we present analytic models that reveal how different document retrieval strategies affect the quality of the extracted relation. Finally, we present our maximum likelihood approach for estimating, on the fly, the parameters required by our analytic models to predict the runtime and the output quality of each execution plan. Our experimental evaluation demonstrates that our optimization approach predicts accurately the output quality and selects the fastest execution plan that satisfies the output quality restrictions.

This paper presents our algorithmic approach for information and relation extraction from unstructured texts (such as from eBook sections or webpages), performing other useful analytics on the text, and automatically generating a semantically meaningful structure (RDF schema). Our algorithmic formulation parses the unstructured text from eBooks and identifies key concepts described in the eBook along with relationship between the concepts. The extracted information is then used for four purposes: (a) for generating some computed metadata about the text source (such as readability of an eBook), (b) generate a concept profile for each distinct part of text, (c) identifying and plotting relationship between key concepts described in the text, and (d) to generate RDF representation for the text source. We have done our experiments on eBook texts from Computer Science domain; however, the approach can be applied to work on different forms of text in other domains as well. The results are not only useful for concept based tagging and navigation of unstructured text documents (such as eBook) but can also be used to design a comprehensive and sophisticated learning recommendation system.

It is an emerging research topic in NLP to joint extraction of knowledge including entities and relations from unstructured text and representing them as meaningful triplets. Despite significant progresses made by recent deep neural network based solutions, these methods still confront the overlapping issue that different relational triplets may have overlapped entities in a sentence, and it is troublesome to address this issue by current solutions. In this paper, we propose a novel multiple tagging model to address the overlapping issue and extract knowledge from unstructured text. Specifically, we devise a multiple tagging scheme that transforms the problem of joint entity and relation extraction into a multiple sequence tagging problem. By using GRU as the building block for encoding-decoding, the proposed model is capable of handling the triplet overlapping problem because the decoder layer allows one entity to take part in more than one triplet. The whole network is end-to-end trianable and outputs all triplets in a sentence directly. Experimental results on the NYT and KBP benchmarks demonstrate that the proposed model siginificantly improves the recall of triplet, and consequently, achieving the new state-of-the-art in the task of triplet extraction on both datasets.

Machine-readable semantic knowledge in the form of taxonomies (i.e., a collection of is-a edges) has proved to be beneficial in an array of NLP tasks including inference, textual entailment, question answering and information extraction. Such widespread utility of taxonomies has led to multiple large-scale manual efforts towards taxonomy induction such as WordNet and Cyc. However, manual construction of taxonomies is time-intensive, and usually, requires substantial annotation efforts by domain experts. Furthermore, the resulting taxonomies suffer from low coverage and are unavailable for specific domains or languages. Therefore, in recent years, there has been a growing body of work, which aims to induce taxonomies automatically, either from semi-structured knowledge resources or unstructured text. In this thesis, we focus on the task of automated taxonomy induction under a variety of different settings. We first focus on taxonomy induction from the largest semi-structured knowledge resource, i.e., Wikipedia. More specifically, we introduce a set of novel heuristics aimed towards inducing a large-scale taxonomy from the English Wikipedia categories network. We also propose a novel comprehensive path-based evaluation framework for taxonomies. Taxonomy induced using our approach significantly outperforms the state of the art across both edge-based as well as path-based evaluation metrics. Moreover, our experiments also demonstrate that high accuracy in edge-based evaluation metrics does not always translate to high-accuracy in path-based evaluation metrics. Subsequently, we propose a novel approach, which leverages the interlanguage links of Wikipedia to induce taxonomies in other Wikipedia languages. Compared to the state of the art, our approach is simpler, more principled, and results in taxonomies that are significantly more accurate across both edge-based and path-based evaluation metrics. In the second part of this thesis, we focus on taxonomy induction from unstructured text. We propose a novel approach towards taxonomy induction from an input vocabulary of seed terms. Unlike previous approaches, which extract singular hypernym edges for terms, our approach utilizes a novel probabilistic framework to extract long-range hypernym subsequences. Taxonomy induction from the extracted subsequences is cast as an instance of the minimum-cost flow optimization problem on a carefully-designed flow network. Through experiments, we demonstrate that our approach outperforms the state of the art across four languages. We also show that our approach is robust to the presence of noise in the input vocabulary. Our approach facilitates the relaxation of many simplifying assumptions, which were employed by previous taxonomy induction approaches. As a result, our work serves to automate the process of taxonomy induction from unstructured text in the true sense. Finally, we introduce a task of discovering and generalizing lexicalized templates from the titles of Wikipedia entities. The experimental results on this task demonstrate that taxonomies, which perform better on our proposed path-based metrics, result in a more accurate set of generalizations for a given set of entities. In summary, this thesis proposes new approaches towards automated taxonomy induction. It improves upon the state of the art in a variety of different settings. It also relaxes many simplifying assumptions that limited the applicability of prior approaches.

The emergence of large language models (LLMs) and the increasing prevalence of electronic health records (EHRs) present significant opportunities for advancing health care research and practice. However, research that compares and applies LLMs to extract key epilepsy-related information from unstructured medical free text is under-explored. This study fills this gap by comparing and applying different open-source LLMs and methods to extract epilepsy information from unstructured clinic letters, thereby optimizing EHRs as a resource for the benefit of epilepsy research. We also highlight some limitations of LLMs. Employing a dataset of 280 annotated clinic letters from King's College Hospital, we explored the efficacy of open-source LLMs (Llama and Mistral series) for extracting key epilepsy-related information, including epilepsy type, seizure type, current anti-seizure medications (ASMs), and associated symptoms. The study used various extraction methods, including direct extraction, summarized extraction, and contextualized extraction, complemented by role-prompting and few-shot prompting techniques. Performance was evaluated against a gold standard dataset, and was also compared to advanced fine-tuned models and human annotations. Llama 2 13b (a 13-billion-parameter LLM developed by Meta) demonstrated superior extraction capabilities across tasks by consistently outperforming other LLMs (F1 = .80 in epilepsy-type extraction, F1 = .76 in seizure-type extraction, and F1 = .90 in current ASMs extraction). Here, F1 score is a balanced metric indicating the model's accuracy in correctly identifying relevant information without excessive false positives. The study highlights the direct extraction showing consistent high performance. Comparative analysis showed that LLMs outperformed current approaches like MedCAT (Medical Concept Annotation Tool) in extracting epilepsy-related information (.2 higher in F1). The results affirm the potential of LLMs in medical information extraction relating to epilepsy, offering insights into leveraging these models for detailed and accurate data extraction from unstructured texts. The study underscores the importance of method selection in optimizing extraction performance and suggests a promising avenue for enhancing medical research and patient care through advanced natural language processing technologies.

The need for naturalistic interaction and communication between humans and machines is essential for high-level artificial intelligence. Matured Natural Language Processing is the first step towards attaining artificial intelligence by automation equal to human beings. It has been a hot and active area of research among researchers for a long. In the advanced forms of natural language processing-based intelligent application building must-have capabilities of generating summaries, understanding context, representing and combining multiple information, and knowledge extraction. Artificial Intelligence and various computing ways (Statistical Reasoning, Machine Learning, and Deep Learning) for achieving it, have been proven the next big weapon for attaining a true sense of intelligent automation and smart system building. Many tasks related to natural language processing have shown consistent and comparable high performances since the introduction of deep learning modelling. Using sequence-to-sequence models has been advantageous and demonstrated promising outcomes in many other areas, including machine translation, speech recognition, picture captioning, etc. This paper aims to propose a novel framework to extract important features from an unstructured text by removing the noisy, irrelevant, and redundant components, and to provide accountability for the information delivered by retrieving image-based text, deep text summarization by knowledge weight calculation, and representation- based on the extracted features.

A wealth of medical knowledge has been encoded in terminologies like SNOMED CT, NCI, FMA, and more. However, these resources are usually lacking information like relations between diseases, symptoms, and risk factors preventing their use in diagnostic or other decision making applications. In this paper we present a pipeline for extracting such information from unstructured text and enriching medical knowledge bases. Our approach uses Semantic Role Labelling and is unsupervised. We show how we dealt with several deficiencies of SRL-based extraction, like copula verbs, relations expressed through nouns, and assigning scores to extracted triples. The system have so far extracted about 120K relations and in-house doctors verified about 5k relationships. We compared the output of the system with a manually constructed network of diseases, symptoms and risk factors build by doctors in the course of a year. Our results show that our pipeline extracts good quality and precise relations and speeds up the knowledge acquisition process considerably.

The power industry usually records equipment failures, defects and other information in the form of text, which contains lots of regular patterns. Knowledge extraction in fault text is of great significance to improve efficiency and reduce the labor cost in the power industry. However, the research for knowledge extraction of text information in this field is rare, it is even more difficult to use machine learning algorithms to mine the deep patterns. To solve this problem, a method of knowledge extraction is proposed in this field. We use power equipment fault texts and relevant guidance as raw materials. Firstly, the knowledge base of this field is designed and constructed based on the ontology concepts, including ontology concept base, description base and regular expression base. Then, the knowledge extraction algorithm is designed according to the knowledge base. After that we conduct the knowledge merge operation to make the extraction results more accurate. Experiments on the real fault texts shows the feasibility and the high accuracy of our method when compared with artificial extraction.

An important approach to text mining involves the use of natural-language information extraction. Information extraction (IE) distills structured data or knowledge from unstructured text by identifying references to named entities as well as stated relationships between such entities. IE systems can be used to directly extricate abstract knowledge from a text corpus, or to extract concrete data from a set of documents which can then be further analyzed with traditional data-mining techniques to discover more general patterns. We discuss methods and implemented systems for both of these approaches and summarize results on mining real text corpora of biomedical abstracts, job announcements, and product descriptions. We also discuss challenges that arise when employing current information extraction technology to discover knowledge in text.

Data mining techniques for extracting knowledge from text have been applied extensively to applications including question answering, document summarisation, event extraction and trend monitoring. However, current methods have mainly been tested on small-scale customised data sets for specific purposes. The availability of large volumes of data and high-velocity data streams (such as social media feeds) motivates the need to automatically extract knowledge from such data sources and to generalise existing approaches to more practical applications. Recently, several architectures have been proposed for what we callknowledge mining: integrating data mining for knowledge extraction from unstructured text (possibly making use of a knowledge base), and at the same time, consistently incorporating this new information into the knowledge base. After describing a number of existing knowledge mining systems, we review the state-of-the-art literature on both current text mining methods (emphasising stream mining) and techniques for the construction and maintenance of knowledge bases. In particular, we focus on mining entities and relations from unstructured text data sources, entity disambiguation, entity linking and question answering. We conclude by highlighting general trends in knowledge mining research and identifying problems that require further research to enable more extensive use of knowledge bases.

Personally Identifiable Information (PII) has gained much attention with the rapid development of technologies and the exploitation of information relating to an individual. The corporates and other organizations store a large amount of information that is primarily disseminated in the form of emails that include personnel information of the user, employee, and customers. The security aspects of PII storage have been ignored, raising serious security concerns onindividual privacy. A significant concern arises about comprehending the responsibilities regarding the uses of PII. However, in real-time scenarios, email data is regarded as unstructured text data, detecting PII from such an unstructured large text corpus is quite challenging. This paper presents an intelligent clustering approach for automatically detecting personally identifiable information (PII) from a large text corpus. The focus of the proposed study is to design a model that receives text content and detects possible PII attributes. Therefore, this paper presents a clustering-based PII Model (C-PPIM) based on NLP and unsupervised learning to address detection of PII in the unstructured large text corpus. NLP is used to perform topic modeling, and Byte mLSTM, a different approach of sequence model, is implemented to address clustering problems in PII detection. The performance analysis of the proposed model is carried out existing hierarchical clustering concerning silhouette and cohesion score. The outcome indicatedthe effectiveness of the proposed system that highlights significant PII attributes, with significant scope in real-time implementation. In contrast, existing techniques are too expensive to function and fit in real-time environments.

Senses are the extensions of language and interrelations between word meanings within language and the basis of flexible language use. Computers are basically made to process symbols. Computational linguistics is to simulate part of, if not all of, the natural language competence. From the perspective of computational linguistics, this paper sheds some light on semantic formalization, and proposes tagging of senses along with tagging weight values when processing word meanings. Also, this paper makes attempts to apply sense weight values to sentence processing in computational linguistics. Keywords-semantics, sense, weight, formalization, computational linguistics I. SEMANTIC KNOWLEDGE IN LANGUAGE PROCESSING According to prototype theory, semantic knowledge is part of human’s cognitive and belief systems. Semantics is also the critical research area since its initiation in 1893 by Michael Breal, and then by Trier and Lakoff who contributes to semantic research from both the descriptive and generative perspective. In linguistic semantics, rules and principles explain how literal meaning is coded in language and how to understand the “senses” of an expression or sense-relations that hold between one word and other expressions within the same language (Ceusters et al, 1999). Language is the carrier of both thoughts and forms. Thoughts are to induce, deduce, infer and judge, while senses are the contents and basis of thoughts. Senses are not the repeated addition of forms, rather a psychological process. Semantics is the mapping of cognitive structure from language expressions. Based on concepts, which are the abstract and general categorization of objects, people can apply semantic principles, situational contexts and encyclopedic knowledge to construct cognitive schema of the outside world. With frequent interactions and cognitive strategies, limited forms can be used to express unlimited connotation of thoughts (Tesser and Leone, 1977). For example, same meanings can be expressed with different semantic forms. a) 他这个人真有意思 (funny); b) 于是人们以为他们有了意思 (wish); c) 你向他意思意思 (express)如何; d) .我根本没有那个意思 (thought); e) 他常说人生真没有意思 (nonsense); f) 你们这么说是什么意思 (intention) These sentences are easy to comprehend. People not only use the grammatical structures but also the sense weights to process the words. The sense weight schema are stored in memory, and activated to represent and distill other sense weights. In this chain activation, sense weights are restructured and created to reflect prototypes. For example, “你好?”, “你好么?”, “好么你?”. Different structures are used in the above examples, but their essential meaning remains the same. So, semantic knowledge is of utmost importance in language comprehension. II. THE STATUS QUO OF COMPUTATIONAL LINGUISTICS Three domains and three corresponding levels of study are involved in building natural language processing systems, i.e. theoretical linguistics and grammar level, computational linguistics and formalization level, computer technology and realization level. Among them, grammar level concerns the innate characteristics of language and paves the way for the latter modules. Realization level focuses on how to control computer operation and provides the efficient development tools and environment. Formalization level is the interface of grammar level and realization level, with the task to readjust the general grammar modules to form grammar modules for convenient computer processing. Computational linguists’ mission is to establish formal grammars (Kaufmann and Pfister, 2012). Computational linguistics is also to establish formalized mathematical modules to analyze and process natural language, and develop computer programs to make it possible that machines can simulate part of, if not all of, human behavior and competence. Computers can only be used to process symbols and all those processed by computers are symbolized with no exception of languages (Yushi, 2003). Meanwhile, there have been breakthroughs in processing Chinese characters and phrases, so sentence level processing has become critical. Basic researches in syntactic, semantic and pragmatic knowledge are the frontier studies in Chinese processing. And how to acquire syntactic and semantic knowledge is the key, while the latter constitutes the major difficulty in processing the word level senses. As aforementioned, semantic formalization can take on many impossible missions in artificial intelligence and computational linguistics, and face challenges at the same time. Thus, the researcher will use semantic field to investigate formalization issues. Semantic fields are the sememe systems, which include the semantemes to represent similarities and differences, and are classified by the subordinates, i.e. degree and number of abstraction of Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013) Published by Atlantis Press, Paris, France.

This paper explores cognitive place associations; conceptualised as a place‐based mental model that derives subconscious links between geographic locations. Utilising a large corpus of online discussion data from the social media website Reddit, we experiment on the extraction of such geographic knowledge from unstructured text. First we construct a system to identify place names found in Reddit comments, disambiguating each to a set of coordinates where possible. Following this, we build a collective picture of cognitive place associations in the United Kingdom, linking locations that co‐occur in user comments and evaluating the effect of distance on the strength of these associations. Exploring these geographies nationally, associations were shown to be typically weaker over greater distances. This distance decay is also highly regional, rural areas typically have greater levels of distance decay, particularly in Wales and Scotland. When comparing major cities across the UK, we observe distinct distance decay patterns, influenced primarily by proximity to other cities.