A Multi-Hop Graph Reasoning Network for Knowledge-Based VQA

Scene graphs are powerful representations that parse images into their abstract semantic elements, i.e., objects and their interactions, which facilitates visual comprehension and explainable reasoning. On the other hand, commonsense knowledge graphs are rich repositories that encode how the world is structured, and how general concepts interact. In this paper, we present a unified formulation of these two constructs, where a scene graph is seen as an image-conditioned instantiation of a commonsense knowledge graph. Based on this new perspective, we re-formulate scene graph generation as the inference of a bridge between the scene and commonsense graphs, where each entity or predicate instance in the scene graph has to be linked to its corresponding entity or predicate class in the commonsense graph. To this end, we propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them, while gradually refining their bridge in each iteration. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs. Through extensive experimentation, we showcase the superior accuracy of GB-Net compared to the most recent methods, resulting in a new state of the art. We publicly release the source code of our method (https://github.com/alirezazareian/gbnet).

Multi-modal knowledge graphs representation learning via multi-headed self-attention

Knowledge graph technology is a critical part of artificial intelligence research. It establishes a knowledge base with the capacity of semantic processing and open interconnection in order to provide intelligent information service, such as search, question-answering, personalized recommendation, and so on. This article first presents a comprehensive study on definitions and architectures of knowledge graphs. Then we summarizes recent advances in knowledge graphs, including knowledge extraction, knowledge representation, knowledge fusion, and knowledge reasoning, with typical applications. Finally, this article concludes with future challenges of knowledge graphs.

RuMER-RL: A hybrid framework for sparse knowledge graph explainable reasoning

Clinical decision support system for hypertension medication based on knowledge graph

BackgroundIn Literature-based Discovery (LBD), Swanson’s original ABC model brought together isolated public knowledge statements and assembled them to infer putative hypotheses via logical connections. Modern LBD studies that scale up this approach through automation typically rely on a simple entity-based knowledge graph with co-occurrences and/or semantic triples as basic building blocks. However, our analysis of a knowledge graph constructed for a recent LBD system reveals limitations arising from such pairwise representations, which further negatively impact knowledge inference. Using LBD as the context and motivation in this work, we explore limitations of using pairwise relationships only as knowledge representation in knowledge graphs, and we identify impacts of these limitations on knowledge inference. We argue that enhanced knowledge representation is beneficial for biological knowledge representation in general, as well as for both the quality and the specificity of hypotheses proposed with LBD.ResultsBased on a systematic analysis of one co-occurrence-based LBD system focusing on Alzheimer’s Disease, we identify 7 types of limitations arising from the exclusive use of pairwise relationships in a standard knowledge graph—including the need to capture more than two entities interacting together in a single event—and 3 types of negative impacts on knowledge inferred with the graph—Experimentally infeasible hypotheses, Literature-inconsistent hypotheses, and Oversimplified hypotheses explanations. We also present an indicative distribution of different types of relationships. Pairwise relationships are an essential component in representation frameworks for knowledge discovery. However, only 20% of discoveries are perfectly represented with pairwise relationships alone. 73% require a combination of pairwise relationships and nested relationships. The remaining 7% are represented with pairwise relationships, nested relationships, and hypergraphs.ConclusionWe argue that the standard entity pair-based knowledge graph, while essential for representing basic binary relations, results in important limitations for comprehensive biological knowledge representation and impacts downstream tasks such as proposing meaningful discoveries in LBD. These limitations can be mitigated by integrating more semantically complex knowledge representation strategies, including capturing collective interactions and allowing for nested entities. The use of more sophisticated knowledge representation will benefit biological fields with more expressive knowledge graphs. Downstream tasks, such as LBD, can benefit from richer representations as well, allowing for generation of implicit knowledge discoveries and explanations for disease diagnosis, treatment, and mechanism that are more biologically meaningful.

Cyberlearning is challenged by the lack of personal and assessment-driven learning, and students are often puzzled by the lack of instructor guidance and feedback, the huge volume and diversity of the learning materials, and thelack of the ability to zoom in from the general concepts to the more specific ones, or the opposite. Intelligent tutoring systems are needed to improve the cyberlearning quality. One of the major difficulties is knowledge representation. The current industry standard is to use Web Ontology Language (OWL) forrepresenting knowledge structure. But OWL only supports one first-class relation, is-a, between the concepts, and different knowledge areas usually need different custom relations to describe the relations among the concepts. For example part-of and time dependency are important relations torepresent most engineering knowledge bodies. OWL has to use object properties to emulate such custom relations, leading to awkward knowledge representation hard for domain experts to code, validate and use such knowledge bases. This research uses Pace University's extension to OWL, named Knowledge Graph (KG), to support knowledge representation with custom relations. The instructors can use Pace University extended Protege IDE to declare and apply custom relations in a single document. The instructor teaching experience is also coded in the KG to better support custom learning order by students with different backgrounds. The prototype of a knowledge-driven tutoring system was designed and implemented to illustrate how the KG supports integrated assessments, using assessment results to custom student learning order or material, and let the students freely navigate in the knowledge space from general to specific or the opposite, and following various custom relations. A web technology tutorial is used to validate the design and effectiveness of this approach.

The goal of the question-answering (QA) system is to understand the questions from users and return their accurate answers. In the medical field, the question-answering system amis to understand patients' questions and return the correct answers. The existed knowledge base question-answering (KB-QA) systems mainly rely on hand-crafted features and ignore structure information of knowledge base which accordingly lead to the answers with low accuracy. In this paper, a novel KB-QA model is put forward based on knowledge representation and recurrent convolutional neural network. This model has three parts, candidate answers generation, entity relationships extraction and knowledge representation learning based on knowledge base. In addition, an algorithm is also developed to compute the scores of linking candidate answers and knowledge base. Experimental results show that the presented model achieves better performance compared with the baseline systems.

Knowledge-based question answering (KBQA) is an essential but challenging task for artificial intelligence and natural language processing. A key challenge pertains to the design of effective algorithms for relation detection. Conventional methods model questions and candidate relations separately through the knowledge bases (KBs) without considering the rich word-level interactions between them. This approach may result in local optimal results. This article presents a global–local attentive relation detection model (GLAR) that utilizes the local module to learn the features of word-level interactions and employs the global module to acquire nonlinear relationships between questions and their candidate relations located in KBs. This article also reports on the application of an end-to-end retrieval-based KBQA system incorporating the proposed relation detection model. Experimental results obtained on two datasets demonstrated GLAR's remarkable performance in the relation detection task. Furthermore, the functioning of end-to-end KBQA systems was significantly improved through the relation detection model, whose results on both datasets outperformed even state-of-the-art methods. Impact Statement —Knowledge-based question answering (KBQA) aims at answering user questions posed over the knowledge bases (KBs). KBQA helps users access knowledge in the KBs more easily, and it works on two subtasks: entity mention detection and relation detection. While existing relation detection algorithms perform well on the global representation of questions and relations sequences, they ignore some local semantic information on interaction cases between them. The technology proposed in this article takes both global and local interactions into account. With superior improvement on two relation detection tasks and two KBQA end tasks, the technology provides more precise answers. It could be used in more applications, including intelligent customer service, intelligent finance, and others.

Few‐shot knowledge graph (KG) reasoning is the main focus in the field of knowledge graph reasoning. In order to expand the application fields of the knowledge graph, a large number of studies are based on a large number of training samples. However, we have learnt that there are actually many missing relationships or entities in the knowledge graph, and in most cases, there are not many training instances when implementing new relationships. To tackle it, in this study, the authors aim to predict a new entity given few reference instances, even only one training instance. A few‐shot learning framework based on a hybrid attention mechanism is proposed. The framework employs traditional embedding models to extract knowledge, and uses an attenuated attention network and a self‐attention mechanism to obtain the hidden attributes of entities. Thus, it can learn a matching metric by considering both the learnt embeddings and one‐hop graph structures. The experimental results present that the model has achieved significant performance improvements on the NELL‐One and Wiki‐One datasets.

Visual commonsense knowledge has received growing attention in the reasoning of long-tailed visual relationships biased in terms of object and relation labels. Most current methods typically collect and utilize external knowledge for visual relationships by following the fixed reasoning path of {subject, object → predicate} to facilitate the recognition of infrequent relationships. However, the knowledge incorporation for such fixed multidependent path suffers from the data set biased and exponentially grown combinations of object and relation labels and ignores the semantic gap between commonsense knowledge and real scenes. To alleviate this, we propose configurable graph reasoning (CGR) to decompose the reasoning path of visual relationships and the incorporation of external knowledge, achieving configurable knowledge selection and personalized graph reasoning for each relation type in each image. Given a commonsense knowledge graph, CGR learns to match and retrieve knowledge for different subpaths and selectively compose the knowledge routed path. CGR adaptively configures the reasoning path based on the knowledge graph, bridges the semantic gap between the commonsense knowledge, and the real-world scenes and achieves better knowledge generalization. Extensive experiments show that CGR consistently outperforms previous state-of-the-art methods on several popular benchmarks and works well with different knowledge graphs. Detailed analyses demonstrated that CGR learned explainable and compelling configurations of reasoning paths.

Walk-based models have shown their advantages in knowledge graph (KG) reasoning by achieving decent performance while providing interpretable decisions. However, the sparse reward signals offered by the KG during a traversal are often insufficient to guide a sophisticated walk-based reinforcement learning (RL) model. An alternate approach is to use traditional symbolic methods (e.g., rule induction), which achieve good performance but can be hard to generalize due to the limitation of symbolic representation. In this paper, we propose RuleGuider, which leverages high-quality rules generated by symbolic-based methods to provide reward supervision for walk-based agents. Experiments on benchmark datasets shows that RuleGuider clearly improves the performance of walk-based models without losing interpretability.

The field of visual question answering (VQA) has seen a growing trend of integrating external knowledge sources to improve performance. However, owing to the potential incompleteness of external knowledge sources and the inherent mismatch between different forms of data, current knowledge-based visual question answering (KBVQA) techniques are still confronted with the challenge of effectively integrating and utilizing multiple heterogeneous data. To address this issue, a novel approach centered on a multi-modal semantic graph (MSG) is proposed. The MSG serves as a mechanism for effectively unifying the representation of heterogeneous data and diverse types of knowledge. Additionally, a multi-modal semantic graph knowledge reasoning model (MSG-KRM) is introduced to perform reasoning and deep fusion of image–text information and external knowledge sources. The development of the semantic graph involves extracting keywords from the image object detection information, question text, and external knowledge texts, which are then represented as symbol nodes. Three types of semantic graphs are then constructed based on the knowledge graph, including vision, question, and the external knowledge text, with non-symbol nodes added to connect these three independent graphs and marked with respective node and edge types. During the inference stage, the multi-modal semantic graph and image–text information are embedded into the feature semantic graph through three embedding methods, and a type-aware graph attention module is employed for deep reasoning. The final answer prediction is a blend of the output from the pre-trained model, graph pooling results, and the characteristics of non-symbolic nodes. The experimental results on the OK-VQA dataset show that the MSG-KRM model is superior to existing methods in terms of overall accuracy score, achieving a score of 43.58, and with improved accuracy for most subclass questions, proving the effectiveness of the proposed method.

Inductive logic reasoning is a fundamental task in graph analysis, which aims to generalize patterns from data. This task has been extensively studied for traditional graph representations, such as knowledge graphs (KGs), using techniques like inductive logic programming (ILP). Existing ILP methods assume learning from KGs with static facts and binary relations. Beyond KGs, graph structures are widely present in other applications such as procedural instructions, scene graphs, and program executions. While ILP is beneficial for these applications, applying it to those graphs is nontrivial: they are more complex than KGs, which usually involve timestamps and n-ary relations, effectively a type of hypergraph with temporal events. In this work, we propose temporal inductive logic reasoning (TILR), an ILP method that reasons on temporal hypergraphs. To enable hypergraph reasoning, we introduce the multi-start random B-walk, a novel graph traversal method for hypergraphs. By combining it with a path-consistency algorithm, TILR learns logic rules by generalizing from both temporal and relational data. To address the lack of hypergraph benchmarks, we create and release two temporal hypergraph datasets: YouCook2-HG and nuScenes-HG. Experiments on these benchmarks demonstrate that TILR achieves superior reasoning capability over various strong baselines.

Knowledge-based visual question answering (VQA) involves answering questions that require external knowledge not present in the image. Existing methods first retrieve knowledge from external resources, then reason over the selected knowledge, the input image, and question for answer prediction. However, this two-step approach could lead to mismatches that potentially limit the VQA performance. For example, the retrieved knowledge might be noisy and irrelevant to the question, and the re-embedded knowledge features during reasoning might deviate from their original meanings in the knowledge base (KB). To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by GPT-3’s power in knowledge retrieval and question answering, instead of using structured KBs as in previous work, we treat GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge. Specifically, we first convert the image into captions (or tags) that GPT-3 can understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just providing a few in-context VQA examples. We further boost performance by carefully investigating: (i) what text formats best describe the image content, and (ii) how in-context examples can be better selected and used. PICa unlocks the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa surpasses the supervised state of the art by an absolute +8.6 points on the OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent few-shot performance.