Random Semantic Tensor Ensemble for Scalable Knowledge Graph Link Prediction

Knowledge graph embedding (KGE) models represent each entity and relation of a knowledge graph (KG) with low-dimensional embedding vectors. These methods have recently been applied to KG link prediction and question answering over incomplete KGs (KGQA). KGEs typically create an embedding for each entity in the graph, which results in large model sizes on real-world graphs with millions of entities. For downstream tasks these atomic entity representations often need to be integrated into a multi stage pipeline, limiting their utility. We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for KG link prediction and incomplete KG question answering. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with autoregressive decoding. Such a simple but powerful method reduces the model size up to 98% compared to conventional KGE models while keeping inference time tractable. After finetuning this model on the task of KGQA over incomplete KGs, our approach outperforms baselines on multiple large-scale datasets without extensive hyperparameter tuning.

Top-[formula omitted] star queries on knowledge graphs through semantic-aware bounding match scores

While conversing with chatbots, humans typically tend to ask many questions, a significant portion of which can be answered by referring to large-scale knowledge graphs (KG). While Question Answering (QA) and dialog systems have been studied independently, there is a need to study them closely to evaluate such real-world scenarios faced by bots involving both these tasks. Towards this end, we introduce the task of Complex Sequential QA which combines the two tasks of (i) answering factual questions through complex inferencing over a realistic-sized KG of millions of entities, and (ii) learning to converse through a series of coherently linked QA pairs. Through a labor intensive semi-automatic process, involving in-house and crowdsourced workers, we created a dataset containing around 200K dialogs with a total of 1.6M turns. Further, unlike existing large scale QA datasets which contain simple questions that can be answered from a single tuple, the questions in our dialogs require a larger subgraph of the KG. Specifically, our dataset has questions which require logical, quantitative, and comparative reasoning as well as their combinations. This calls for models which can: (i) parse complex natural language questions, (ii) use conversation context to resolve coreferences and ellipsis in utterances, (iii) ask for clarifications for ambiguous queries, and finally (iv) retrieve relevant subgraphs of the KG to answer such questions. However, our experiments with a combination of state of the art dialog and QA models show that they clearly do not achieve the above objectives and are inadequate for dealing with such complex real world settings. We believe that this new dataset coupled with the limitations of existing models as reported in this paper should encourage further research in Complex Sequential QA.

Besides the challenge that a human can ask one question in many different ways, a key aspect in Question Answering approaches over Knowledge Graphs (KGQA) is to deal with the vast amount of information present in the knowledge graphs. Modern real-world knowledge graphs contain nearly millions of entities and relationships. Additionally, they are enriched with new facts every day. However, not all facts are relevant for answering particular questions, thus fostering several challenges to KGQA systems, which require interpretable and query-able data. One solution to filtering the extra data in knowledge graphs is to rely on graph summarization techniques. Graph-based summarization approaches aim to resize knowledge graphs to be more concise and precise by storing only relevant information. In this paper, we propose a framework named LAUREN that applies different summarization techniques on knowledge graphs to be used in KGQA systems. Our experiments show that LAUREN summarizes large knowledge graphs such as DBpedia by 2 million entities and its summarization still achieves the same performance on both question answering and linking tasks compared to the complete DBpedia.

Improving embedding-based link prediction performance using clustering

Knowledge Graphs (KGs) comprise of interlinked information in the form of entities and relations between them in a particular domain and provide the backbone for many applications. However, the KGs are often incomplete as the links between the entities are missing. Link Prediction is the task of predicting these missing links in a KG based on the existing links. Recent years have witnessed many studies on link prediction using KG embeddings which is one of the mainstream tasks in KG completion. To do so, most of the existing methods learn the latent representation of the entities and relations whereas only a few of them consider contextual information as well as the textual descriptions of the entities. This paper introduces an attentive encoder-decoder based link prediction approach considering both structural information of the KG and the textual entity descriptions. Random walk based path selection method is used to encapsulate the contextual information of an entity in a KG. The model explores a bidirectional Gated Recurrent Unit (GRU) based encoder-decoder to learn the representation of the paths whereas SBERT is used to generate the representation of the entity descriptions. The proposed approach outperforms most of the state-of-the-art models and achieves comparable results with the rest when evaluated with FB15K, FB15K-237, WN18, WN18RR, and YAGO3-10 datasets.

Neuro-Symbolic Artificial Intelligence (AI) focuses on integrating symbolic and sub-symbolic systems to enhance the performance and explainability of predictive models. Symbolic and sub-symbolic approaches differ fundamentally in how they represent data and make use of data features to reach conclusions. Neuro-symbolic systems have recently received significant attention in the scientific community. However, despite efforts in neural-symbolic integration, symbolic processing can still be better exploited, mainly when these hybrid approaches are defined on top of knowledge graphs. This work is built on the statement that knowledge graphs can naturally represent the convergence between data and their contextual meaning (i.e., knowledge). We propose a hybrid system that resorts to symbolic reasoning, expressed as a deductive database, to augment the contextual meaning of entities in a knowledge graph, thus, improving the performance of link prediction implemented using knowledge graph embedding (KGE) models. An entity context is defined as the ego network of the entity in a knowledge graph. Given a link prediction task, the proposed approach deduces new RDF triples in the ego networks of the entities corresponding to the heads and tails of the prediction task on the knowledge graph (KG). Since knowledge graphs may be incomplete and sparse, the facts deduced by the symbolic system not only reduce sparsity but also make explicit meaningful relations among the entities that compose an entity ego network. As a proof of concept, our approach is applied over a KG for lung cancer to predict treatment effectiveness. The empirical results put the deduction power of deductive databases into perspective. They indicate that making explicit deduced relationships in the ego networks empowers all the studied KGE models to generate more accurate links.

Entity representation learning is both fundamental and crucial for link prediction in knowledge graph. Existing entity representation learning approaches mainly focus on learning from knowledge triples with single modal text data which could only describe the things or concepts in words. However, multimodal data commonly exists in the real worlds and may describe things or concepts from different aspects. Therefore, multimodal entities have emerged in knowledge graph in recent years, which greatly enrich the knowledge graph by different information. This paper focuses on how to learn more robust and comprehensive representations of multimodal entities for better link prediction. A new representation learning method for multimodal entities is proposed, called Graph Attention with Bi-pooling Representation Learning (GBRL), where multimodal fusion and knowledge embedding are combined for learning representations of an entity with consideration of its multimodal neighbors. Firstly, feature vectors are obtained by pre-trained models for each modality. Then, the bilinear pooling is utilized for combining feature vectors of multimodal entities into multimodal feature vectors. A two-layer graph attention module is followed to generate multimodal representations of an entity from its multimodal neighbors. Finally, the inception decoder uses multimodal knowledge embedding to calculate scores for link prediction. The experimental results have shown that the proposed method performs well on link prediction by consideration of multimodal entities comparing with traditional methods that using only single modal text data.

Existing methods for knowledge graph embedding do not ensure the high-rank triples predicted by themselves to be as consistent as possible with the logical background which is made up of a knowledge graph and a logical theory. Users must take great effort to filter consistent triples before adding new triples to the knowledge graph. To alleviate users’ burden, we propose an approach to enhancing existing embedding-based methods to encode logical consistency into the learnt distributed representation for the knowledge graph, enforcing high-rank new triples as consistent as possible. To evaluate this approach, four knowledge graphs with logical theories are constructed from the four great classical masterpieces of Chinese literature. Experimental results on these datasets show that our approach is able to guarantee high-rank triples as consistent as possible while preserving a comparable performance as baseline methods in link prediction and triple classification.

Knowledge graphs play an important role in many applications, such as link prediction and question answering. Translating embedding for knowledge graphs is done with the aim of encoding structured information on entities and their rich relations in a low-dimensional embedding space. TransE is one of the most important methods in translation-based models, and uses translation invariance to implement translating embedding for knowledge graphs. In this line of work, translating embedding models represent the relation as a translation from the head entity to the tail entity and have achieved impressive results. Currently, the TransE model is only developed on single-node machines. Unfortunately, the computing and storage capacities of a single machine can easily reach their limits as knowledge graphs become larger and more complex, which limits the application scope of TransE. In order to solve this problem, we propose a distributed TransE method, known as DTransE, which can utilize distributed computing resources to calculate knowledge graph embeddings. However, building a distributed TransE is complicated and involves challenges of knowledge graph partitioning and computation. To solve these challenges, we provide a high-quality edge partitioning algorithm for the power-law graph by considering the high-degree and low-degree vertices with adaptive weights, which can balance the workload. By using the unactivated Gather-Apply-Scatter model on TransE, the processes periodically exchange messages in a loop. The irregular data distribution among the processes is also optimized to further accelerate communication. As far as we know, this is the first work on a distributed TransE method. We use link prediction to evaluate the DTransE in a distributed environment. Experiments show that, compared to the original TransE method, our proposed DTransE is, on average, 24.5 times faster with a minimum loss of accuracy; compared to the state-of-the-art parallel TransE implementation, DTransE is two times faster on average.

Link prediction is an important task in many domains, including health sciences, biology, recommender systems, social networks, and many more. It is one of the problems residing within the intersection of knowledge graphs and machine learning. Link prediction aims to discover unknown links between entities in a graph using various techniques. However, due to the size of knowledge graphs today and their complexity, it is a challenging and time-consuming task. In this work, we present RegPattern2Vec, a method to effectively sample a large knowledge graph to learn node embeddings, while capturing the semantic relationships between graph nodes with minimum prior knowledge and human involvement. Our results show that the link prediction using RegPattern2Vec outperforms related graph embedding approaches on large-scale and complex knowledge graphs.

Incompleteness of large knowledge graphs (KG) has motivated many researchers to propose methods to automatically find missing edges in KGs. A promising approach for KG completion (link prediction) is embedding a KG into a continuous vector space. There are different methods in the literature that learn a continuous representation of KG (latent features of KG). The benchmark dataset FB15k has been widely employed to evaluate these methods. However, It has been noted that FB15k contains many pairs of edges in which a pair represents the same relationship in reverse directions. Therefore, the inverse of numerous test triples occurs in the training set. To address this problem, FB15k-237, a subset of FB15k, was created by removing those inverse-duplicate relations to form a more challenging, realistic dataset. There is not any study that investigates how the aforementioned bias in this widely used benchmark dataset affects the results of embedding-based knowledge graph completion methods and whether their promising results are largely due to the bias. Motivated by this question, we conducted extensive experiments and report the link prediction results on FB15K and FB15k-237 using several embedding-based methods. We compare the results of different methods to see how their performances change in absence of inverse relations. Our experiment results demonstrate that the performance of embedding models in link prediction task diminishes tremendously when the inverse relationships do not exist anymore.

A variety of knowledge graph embedding approaches have been developed. Most of them obtain embeddings by learning the structure of the knowledge graph within a link prediction setting. As a result, the embeddings reflect only the structure of a single knowledge graph, and embeddings for different knowledge graphs are not aligned, e.g., they cannot be used to find similar entities across knowledge graphs via nearest neighbor search. However, knowledge graph embedding applications such as entity disambiguation require a more global representation, i.e., a representation that is valid across multiple sources. We propose to learn universal knowledge graph embeddings from large-scale interlinked knowledge sources. To this end, we fuse large knowledge graphs based on the owl:sameAs relation such that every entity is represented by a unique identity. We instantiate our idea by computing universal embeddings based on DBpedia and Wikidata yielding embeddings for about 180 million entities, 15 thousand relations, and 1.2 billion triples. We believe our computed embeddings will support the emerging field of graph foundation models. Moreover, we develop a convenient API to provide embeddings as a service. Experiments on link prediction suggest that universal knowledge graph embeddings encode better semantics compared to embeddings computed on a single knowledge graph. For reproducibility purposes, we provide our source code and datasets open access.

Multi-modal knowledge graphs representation learning via multi-headed self-attention

Reasoning over knowledge graphs (KGs) has received increasing attention recently due to its promising applications in many areas, such as semantic search and recommendation systems. Subsequently, most reasoning models are inherently transductive and ignore uncertainties of KGs, making it difficult to generalize to unseen entities. Moreover, existing approaches usually require each entity in the KG to have sufficient training samples, which leads to the overfitting of the entity having few instances. In fact, long-tail distributions are quite widespread in KGs, and newly emerging entities will tend to have only a few related triples. In this work, we aim at studying knowledge graph reasoning under a challenging setting where only limited training samples are available. Specifically, we propose a Bayesian inductive reasoning method and incorporate meta-learning techniques in few-shot learning to solve data deficiency and uncertainties. We design a Bayesian graph neural network as a meta-learner to achieve Bayesian inference, which can extrapolate meta-knowledge from observed KG to emerging entities. We conduct extensive experiments on two large-scale benchmark datasets, and the results demonstrate considerable performance improvement with the proposed approach over other baselines.