A novel knowledge enhancement method for large-scale natural language training model

The field of visual question answering (VQA) has seen a growing trend of integrating external knowledge sources to improve performance. However, owing to the potential incompleteness of external knowledge sources and the inherent mismatch between different forms of data, current knowledge-based visual question answering (KBVQA) techniques are still confronted with the challenge of effectively integrating and utilizing multiple heterogeneous data. To address this issue, a novel approach centered on a multi-modal semantic graph (MSG) is proposed. The MSG serves as a mechanism for effectively unifying the representation of heterogeneous data and diverse types of knowledge. Additionally, a multi-modal semantic graph knowledge reasoning model (MSG-KRM) is introduced to perform reasoning and deep fusion of image–text information and external knowledge sources. The development of the semantic graph involves extracting keywords from the image object detection information, question text, and external knowledge texts, which are then represented as symbol nodes. Three types of semantic graphs are then constructed based on the knowledge graph, including vision, question, and the external knowledge text, with non-symbol nodes added to connect these three independent graphs and marked with respective node and edge types. During the inference stage, the multi-modal semantic graph and image–text information are embedded into the feature semantic graph through three embedding methods, and a type-aware graph attention module is employed for deep reasoning. The final answer prediction is a blend of the output from the pre-trained model, graph pooling results, and the characteristics of non-symbolic nodes. The experimental results on the OK-VQA dataset show that the MSG-KRM model is superior to existing methods in terms of overall accuracy score, achieving a score of 43.58, and with improved accuracy for most subclass questions, proving the effectiveness of the proposed method.

Textual entailment is a fundamental task in natural language processing. Most approaches for solving this problem use only the textual content present in training data. A few approaches have shown that information from external knowledge sources like knowledge graphs (KGs) can add value, in addition to the textual content, by providing background knowledge that may be critical for a task. However, the proposed models do not fully exploit the information in the usually large and noisy KGs, and it is not clear how it can be effectively encoded to be useful for entailment. We present an approach that complements text-based entailment models with information from KGs by (1) using Personalized PageRank to generate contextual subgraphs with reduced noise and (2) encoding these subgraphs using graph convolutional networks to capture the structural and semantic information in KGs. We evaluate our approach on multiple textual entailment datasets and show that the use of external knowledge helps the model to be robust and improves prediction accuracy. This is particularly evident in the challenging BreakingNLI dataset, where we see an absolute improvement of 5-20% over multiple text-based entailment models.

We present an online multilingual system for event detection and comprehension from media feeds. The system retrieves information from news sites, aggregates them into events (event detection), and summarizes them by extracting semantic labels of its most relevant entities (event representation) in order to answer the journalism Ws: who, what, when and where. The generated events populate VLX-Stories -an event ontology- transforming unstructured text data to a structured knowledge base representation. Our system exploits an external entity Knowledge Graph (VKG) to help populate VLX-Stories. At the same time, this external knowledge graph can also be extended with a Dynamic Entity Linking (DEL) module, which detects emerging entities (EE) on unstructured data. The system is currently deployed in production and used by media producers in the editorial process, providing real-time access to breaking news. Each month, VLX-Stories detects over 9000 events from over 4000 news feeds from seven different countries and in three different languages. At the same time, it detects over 1300 EE per month, which populate VKG.

Construct a knowledge graph is time-consuming and the knowledge graph in the scientific domain requires extremely high labor costs due to it requires high prior knowledge to extract knowledge from resources. To build a scientific research knowledge graph, the most of input are papers, patent, the description of their project and some national program (such as National High Technology Research and Development Program of China, Major State Basic Research Development Program of China, General Program, Key Program and Major Program) which all of them are unstructured data, that make human participation are mostly necessary to measure the quality. In this paper, we design and proposed a framework using active learning; this framework can be used to extract entity and relation from unstructured science and technology research data. This framework combines the human and machine learning approach together, which is active learning, to help user extract entity from those unstructured data with less time cost. By using those data to construct a CKG as annotation label, it further implements active learning tools and helps the expert to rapidly annotate the data with high accuracy. Those knowledge graph constructed by this framework can be used to finding similar research area, finding similar researchers, finding popular research areas and so on.

This study aims to develop high-performing Machine Learning and Deep Learning models in predicting hospital length of stay (LOS) while enhancing interpretability. We compare performance and interpretability of models trained only on structured tabular data with models trained only on unstructured clinical text data, and on mixed data. The structured data was used to train fourteen classical Machine Learning models including advanced ensemble trees, neural networks and k-nearest neighbors. The unstructured data was used to fine-tune a pre-trained Bio Clinical BERT Transformer Deep Learning model. The structured and unstructured data were then merged into a tabular dataset after vectorization of the clinical text and a dimensional reduction through Latent Dirichlet Allocation. The study used the free and publicly available Medical Information Mart for Intensive Care (MIMIC) III database, on the open AutoML Library AutoGluon. Performance is evaluated with respect to two types of random classifiers, used as baselines. The best model from structured data demonstrates high performance (ROC AUC = 0.944, PRC AUC = 0.655) with limited interpretability, where the most important predictors of prolonged LOS are the level of blood urea nitrogen and of platelets. The Transformer model displays a good but lower performance (ROC AUC = 0.842, PRC AUC = 0.375) with a richer array of interpretability by providing more specific in-hospital factors including procedures, conditions, and medical history. The best model trained on mixed data satisfies both a high level of performance (ROC AUC = 0.963, PRC AUC = 0.746) and a much larger scope in interpretability including pathologies of the intestine, the colon, and the blood; infectious diseases, respiratory problems, procedures involving sedation and intubation, and vascular surgery. Our results outperform most of the state-of-the-art models in LOS prediction both in terms of performance and of interpretability. Data fusion between structured and unstructured text data may significantly improve performance and interpretability.

A knowledge graph method for hazardous chemical management: Ontology design and entity identification

With the development of technology, Earth Science has entered a new era. Continuous research has generated a large amount of Earth Science data, including a significant amount of semi-structured and unstructured data, which contain information about locations, geographical concepts, geological characteristics of mineral deposits, and relationships. Efficient management of these Earth Science data is crucial for the development of digital earth systems, rational planning of resource industries, and resource security. By representing entities, relationships, and attributes through graph structures, knowledge graphs capture and present concepts and facts about the real world, facilitating efficient data management. However, due to the highly specialized and complex nature of Earth Science data and disciplinary differences, the methods used to construct general-purpose knowledge graphs cannot be directly applied to building knowledge graphs in the field of geological science. Therefore, this paper summarizes a “pipeline” approach to constructing an Earth Science knowledge graph in order to clarify the complete construction process and reduce barriers between data and technology. This approach divides the construction of the Earth Science knowledge graph into two parts and designs functional modules under each part to specify the construction process of the knowledge graph. In addition to proposing this approach, a knowledge graph of iron ore deposits is automatically constructed by integrating geographic and geological data related to iron ore deposits using deep learning techniques. The systematic approach presented in this paper reduces the threshold for constructing geological science knowledge graphs, provides methodological support for specific disciplines or research objects in Earth Science, and also lays the foundation for the construction of large-scale Earth Science knowledge graphs that combine crowdsourcing and expert decision-making, as well as the development of intelligent question-answering systems and intelligent decision-making systems covering the entire field of Earth Science.

Decision-making tasks usually follow five steps: identifying the problem, collecting data, extracting evidence, iden-tifying arguments, and making the decision. This paper focuses on two steps of decision-making: extracting evidence by building knowledge graphs (KGs) of specialized topics and identifying sentences' arguments through sentence-level argument mining. We present a hybrid model that combines topic modeling using latent Dirichlet allocation (LDA) and word embeddings to obtain external knowledge from structured and unstructured data. We use a topic model to extract topic- and sentence-specific evidence from the structured knowledge base Wikidata. A knowledge graph is constructed based on the cosine similarity between the entity word vectors of Wikidata and the vector of the given sentence. A second graph based on topic-specific articles found via Google supplements the general incompleteness of the structured knowledge base. Combining these graphs, we obtain a graph-based model that, as our evaluation shows, successfully capitalizes on both structured and unstructured data.

Background Statins are guideline-recommended medications that reduce cardiovascular events in patients with diabetes. Yet, statin use is concerningly low in this high-risk population. Identifying reasons for statin nonuse, which are typically described in unstructured electronic health record data, can inform targeted system interventions to improve statin use. We aimed to leverage a deep learning approach to identify reasons for statin nonuse in patients with diabetes. Methods and Results Adults with diabetes and no statin prescriptions were identified from a multiethnic, multisite Northern California electronic health record cohort from 2014 to 2020. We used a benchmark deep learning natural language processing approach (Clinical Bidirectional Encoder Representations from Transformers) to identify statin nonuse and reasons for statin nonuse from unstructured electronic health record data. Performance was evaluated against expert clinician review from manual annotation of clinical notes and compared with other natural language processing approaches. Of 33 461 patients with diabetes (mean age 59±15 years, 49% women, 36% White patients, 24% Asian patients, and 15% Hispanic patients), 47% (15 580) had no statin prescriptions. From unstructured data, Clinical Bidirectional Encoder Representations from Transformers accurately identified statin nonuse (area under receiver operating characteristic curve [AUC] 0.99 [0.98-1.0]) and key patient (eg, side effects/contraindications), clinician (eg, guideline-discordant practice), and system reasons (eg, clinical inertia) for statin nonuse (AUC 0.90 [0.86-0.93]) and outperformed other natural language processing approaches. Reasons for nonuse varied by clinical and demographic characteristics, including race and ethnicity. Conclusions A deep learning algorithm identified statin nonuse and actionable reasons for statin nonuse in patients with diabetes. Findings may enable targeted interventions to improve guideline-directed statin use and be scaled to other evidence-based therapies.

The International Classification of Diseases (ICD) serves as a widely employed framework for assigning diagnosis codes to electronic health records of patients. These codes facilitate the encapsulation of diagnoses and procedures conducted during a patient’s hospitalisation. This study aims to devise a predictive model for ICD codes based on the MIMIC-III clinical text dataset. Leveraging natural language processing techniques and deep learning architectures, we constructed a pipeline to distill pertinent information from the MIMIC-III dataset: the Medical Information Mart for Intensive Care III (MIMIC-III), a sizable, de-identified, and publicly accessible repository of medical records. Our method entails predicting diagnosis codes from unstructured data, such as discharge summaries and notes encompassing symptoms. We used state-of-the-art deep learning algorithms, such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, bidirectional LSTM (BiLSTM) and BERT models after tokenizing the clinical test with Bio-ClinicalBERT, a pre-trained model from Hugging Face. To evaluate the efficacy of our approach, we conducted experiments utilizing the discharge dataset within MIMIC-III. Employing the BERT model, our methodology exhibited commendable accuracy in predicting the top 10 and top 50 diagnosis codes within the MIMIC-III dataset, achieving average accuracies of 88% and 80%, respectively. In comparison to recent studies by Biseda and Kerang, as well as Gangavarapu, which reported F1 scores of 0.72 in predicting the top 10 ICD-10 codes, our model demonstrated better performance, with an F1 score of 0.87. Similarly, in predicting the top 50 ICD-10 codes, previous research achieved an F1 score of 0.75, whereas our method attained an F1 score of 0.81. These results underscore the better performance of deep learning models over conventional machine learning approaches in this domain, thus validating our findings. The ability to predict diagnoses early from clinical notes holds promise in assisting doctors or physicians in determining effective treatments, thereby reshaping the conventional paradigm of diagnosis-then-treatment care. Our code is available online.

Until recently, structured (e.g., relational) and unstructured (e.g., textual) data were managed very differently: Structured data was queried declaratively using languages such as SQL, while unstructured data was searched using boolean queries over inverted indices. Today, we witness the rapid emergence of Big Data Integration techniques leveraging knowledge graphs to bridge the gap between different types of contents and integrate both unstructured and structured information more effectively. I will start this talk by giving a few examples of Big Data Integration. I will then describe two recent systems built in my lab and leveraging such techniques: ZenCrowd, a socio-technical platform that automatically connects Web documents to semi-structured entities in a knowledge graph, and Guider, a Big Data Integration system for the cloud.

Entity typing in knowledge graphs (KGs) aims to infer missing types of entities and might be considered one of the most significant tasks of knowledge graph construction since type information is highly relevant for querying, quality assurance, and KG applications. While supervised learning approaches for entity typing have been proposed, they require large amounts of (manually) labeled data, which can be expensive to obtain. In this paper, we propose a novel approach for KG entity typing that leverages semi-supervised learning from massive unlabeled data. Our approach follows a teacher-student paradigm that allows combining a small amount of labeled data with a large amount of unlabeled data to boost performance. We conduct several experiments on two benchmarking datasets (FB15k-ET and YAGO43k-ET). Our results demonstrate the effectiveness of our approach in improving entity typing in KGs. Given type information for only 1% of entities, our approach ASSET predicts missing types with a F1-score of 0.47 and 0.64 on the datasets FB15k-ET and YAGO43k-ET, respectively, outperforming supervised baselines.

Rumor posts have received substantial attention with the rapid development of online and social media platforms. The automatic detection of rumor from posts has emerged as a major concern for the general public, the government, and social media platforms. Most existing methods focus on the linguistic and semantic aspects of posts content, while ignoring knowledge entities and concepts hidden within the article which facilitate rumor detection. To address these limitations, in this paper, we propose a novel end-to-end attention and graph-based neural network model (KAGN), which incorporates external knowledge from the knowledge graphs to detect rumor. Specifically, given the post's sparse and ambiguous semantics, we identify entity mentions in the post’s content and link them to entities and concepts in the knowledge graphs, which serve as complementary semantic information for the post text. To effectively inject external knowledge into textual representations, we develop a knowledge-aware attention mechanism to fuse local knowledge. Additionally, we construct a graph consisting of posts texts, entities, and concepts, which is fed to graph convolutional networks to explore long-range knowledge through graph structure. Our proposed model can therefore detect rumor by combining semantic-level and knowledge-level representations of posts. Extensive experiments on four publicly available real-world datasets show that KAGN outperforms or is comparable to other state-of-the-art methods, and also validate the effectiveness of knowledge.

Short text classification has provoked a vast amount of attention and research in recent decades. However, most existing methods only focus on the short texts that contain dozens of words like Twitter and Microblog, while pay far less attention to the extreme short texts like news headline and search snippets. Meanwhile, contemporary short text classification methods that extend the features via external knowledge sources always introduce lots of useless concepts, which may be detrimental to classification performance. Moreover, unlike traditional short text classification methods, the classification results of extreme short texts are often determined by a few even one or two keywords. To address these problems, we propose a novel hybrid classification method via Keywords Screening and Attention Mechanisms in extreme short text, called KSAM. More specifically, firstly, the attention-based BiLSTM is introduced in our method to enhance the role of keywords. Secondly, we screen the keywords in the extreme short text for obtaining the true class label, and the concepts concerning the keywords are retrieved from external open knowledge sources like DBpedia. Thirdly, the attention mechanisms are introduced to acquire the weight of these retrieved concepts. Finally, conceptual information is utilized to assist the classification of the extreme short text. Extensive experiments have demonstrated the effectiveness of our method compared to other state-of-the-art methods.

International knowledge is a key intangible resource for international entrepreneurship. Yet, given the liabilities of newness and foreignness, how do new ventures obtain international knowledge? In this study, we draw on the knowledge‐based view to investigate both internal and external sources of international knowledge for 206 new ventures. The findings suggest international knowledge may be sourced externally, including from alliance partners, venture capital firms, and firms in close proximity. Interestingly, and contrary to absorptive capacity arguments, new ventures with limited internationally experienced top management teams benefited most from external international knowledge sources. Copyright © 2009 Strategic Management Society.