- Research Article
9
- 10.1016/j.imavis.2021.104316
Aligning vision-language for graph inference in visual dialog
- Oct 12, 2021
- Image and Vision Computing
- Tianling Jiang + 4 more +4
Aligning vision-language for graph inference in visual dialog
Visual dialog with semantic consistency: An external knowledge-driven approach.
Aligning vision-language for graph inference in visual dialog
Aligning vision-language for graph inference in visual dialog
GoG: Relation-aware Graph-over-Graph Network for Visual Dialog
Visual dialog, which aims to hold a meaningful conversation with humans about a given image, is a challenging task that requires models to reason the complex dependencies among visual content, dialog history, and current questions. Graph neural networks are recently applied to model the implicit relations between objects in an image or dialog. However, they neglect the importance of 1) coreference relations among dialog history and dependency relations between words for the question representation; and 2) the representation of the image based on the fully represented question. Therefore, we propose a novel relation-aware graph-over-graph network (GoG) for visual dialog. Specifically, GoG consists of three sequential graphs: 1) H-Graph, which aims to capture coreference relations among dialog history; 2) History-aware Q-Graph, which aims to fully understand the question through capturing dependency relations between words based on coreference resolution on the dialog history; and 3) Questionaware I-Graph, which aims to capture the relations between objects in an image based on fully question representation. As an additional feature representation module, we add GoG to the existing visual dialogue model. Experimental results show that our model outperforms the strong baseline in both generative and discriminative settings by a significant margin.
Read moreYou should know more: Learning external knowledge for visual dialog
You should know more: Learning external knowledge for visual dialog
Heterogeneous Knowledge Network for Visual Dialog
Visual dialog requires an agent to answer successive questions considering an image and dialog history, which is a classic vision-language task. Despite progress, there are still two key challenges: 1) parsing long or complex questions and answers and 2) dealing with the visual scene containing complicated interactions among entities. These challenges bring about the unsatisfactory consequence of current visual dialog methods. In this paper, we propose a novel Heterogeneous Knowledge Network (HKNet), which leverages textual sequence knowledge and graph knowledge to address the above issues. Specifically, the textual sequence knowledge is derived from the sentences that are retrieved from the image captions of the visual dialog dataset. The textual sequence knowledge can supplement essential common sense for parsing long or complex questions and answers. The graph knowledge is constructed via scene graph, which provides complete visual relationships for understanding the complicated interactions. These two kinds of heterogeneous knowledge complement each other and jointly improve the logical reasoning ability of the visual dialog. Extensive experimental results on two benchmark datasets: VisDial v0.9 and v1.0 demonstrate the superiority of the proposed HKNet. Ablation studies and visualization results further verify the effectiveness of our method.
Read moreSKANet: Structured Knowledge-Aware Network for Visual Dialog
Visual dialog aims to generate an answer to each question based on an image and dialog history. Despite recent progress, existing methods still undergo degradation on the condition of complex scenarios. Handling these scenarios depends on logical reasoning that requires common sense priors. In this paper, we propose a novel visual dialog pipeline, named Structured Knowledge-Aware Network (SKANet), consisting of a Multi-Modality Fusion Module, an Image Knowledge-Aware Module, and a Caption Knowledge-Aware Module. The Multi-Modality Fusion Module explores the textual context about the dialog history and visual content. To deal with the complex scenarios, the Image and Caption Knowledge-Aware Modules construct common sense knowledge graphs from ConceptNet. Experimental results on the VisDial v1.0 dataset show that our proposed method effectively outperforms comparative methods.
Read moreVisual Dialog.
We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being sufficiently grounded in vision to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person real-time chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial v0.9 has been released and consists of dialog question-answer pairs from 10-round, human-human dialogs grounded in images from the COCO dataset.
Read moreVisual Dialog
We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being grounded in vision enough to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial contains 1 dialog (10 question-answer pairs) on ~140k images from the COCO dataset, with a total of ~1.4M dialog question-answer pairs. We introduce a family of neural encoder-decoder models for Visual Dialog with 3 encoders (Late Fusion, Hierarchical Recurrent Encoder and Memory Network) and 2 decoders (generative and discriminative), which outperform a number of sophisticated baselines. We propose a retrieval-based evaluation protocol for Visual Dialog where the AI agent is asked to sort a set of candidate answers and evaluated on metrics such as mean-reciprocal-rank of human response. We quantify gap between machine and human performance on the Visual Dialog task via human studies. Our dataset, code, and trained models will be released publicly at https://visualdialog.org. Putting it all together, we demonstrate the first visual chatbot!.
Read moreVisual Dialog: Towards Communicative Visual Agents
Recent years have seen significant advancements in artificial intelligence (AI). Still, we are far from intelligent agents that can visually perceive their surroundings, reason, and interact with humans in natural language, therebybeing an integral part of our lives. As a step towards such a grand goal, this thesis proposes Visual Dialog that requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in the image, infer context from history, and answer the questionaccurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence while being grounded in vision enough to allow objective evaluation of responses. We collect VisDial, a large dataset of human dialogs on real images to benchmark progress. In order to tackle several challenges in Visual Dialog, we build neural network-based models that reason about the inputs explicitly by performing visual coreference resolution called CorefNMN. To demonstrate the effectiveness of such models, we test them on the VisDial dataset and a large diagnostic dataset, CLEVR-Dialog, which we synthetically generate with fully annotateddialog states. By breaking down the performance of these models according to history dependency, coreference distance, etc., we show that our models quantitativelyoutperform other approaches and are qualitatively more interpretable, grounded, and consistent—all of which are desirable for an AI system. We then apply Visual Dialog to visually-grounded, goal-driven dialog without the need for additional supervision. Goal-driven dialog agents specialize in a downstream task (or goal) making them deployable and interesting to study. We also train these agents from scratch to solely maximize their performance (goal rewards) and study the emergent language. Our findings show that while most agent-invented language is effective (i.e., achieve near-perfect task rewards), they are decidedly not interpretable or compositional. All our datasets and code are publicly available to encourage future research.
Read moreImage-Collection Summarization Using Scene-Graph Generation With External Knowledge
Summarization tasks aim to summarize multiple pieces of information into a short description or representative information. A text summarization task is a task that summarizes textual information into a short description, whereas in an image collection summarization task, also known as the photo album summarization task, the goal is to find the representative visual information of all images in the collection. In recent years, scene-graph generation has shown the advantage of describing the visual contexts of a single-image, and incorporating external knowledge into the scene-graph generation model has also given effective directions for unseen single-image scene-graph generation. Following this trend, in this paper, we propose a novel scene-graph-based image-collection summarization model. The key idea of the proposed method is to enhance the relation predictor toward relationships between images in an image collection incorporating knowledge graphs as external knowledge for training a model. To evaluate the proposed method, we build an extended annotated MS-COCO dataset for this task and introduce an evaluation process that focuses on estimating the similarity between a summarized scene graph and ground-truth scene graphs. Traditional evaluation focuses on calculating precision and recall scores, which involve true positive predictions without balancing precision and recall. Meanwhile, the proposed evaluation process focuses on calculating the F-score of the similarity between a summarized scene graph and ground-truth scene graphs which aims to balance both false positives and false negatives. Experimental results show that the use of external knowledge in enhancing the relation predictor achieves better results compared with existing methods.
Read moreImproving Cross-Modal Understanding in Visual Dialog Via Contrastive Learning
Visual Dialog is a challenging vision-language task since the visual dialog agent needs to answer a series of questions after reasoning over both the image content and dialog history. Though existing methods try to deal with the cross-modal understanding in visual dialog, they are still not enough in ranking candidate answers based on their understanding of visual and textual contexts. In this paper, we analyze the cross-modal understanding in visual dialog based on the vision-language pre-training model VD-BERT and propose a novel approach to improve the cross-modal understanding for visual dialog, named ICMU. ICMU enhances cross-modal understanding by distinguishing different pulled inputs (i.e. pulled images, questions or answers) based on four-way contrastive learning. In addition, ICMU exploits the single-turn visual question answering to enhance the visual dialog model’s cross-modal understanding to handle a multi-turn visually-grounded conversation. Experiments show that the proposed approach improves the visual dialog model’s cross-modal understanding and brings satisfactory gain to the Vis-Dial dataset.
Read moreCommonsense-Guided Semantic and Relational Consistencies for Image-Text Retrieval
Image-text retrieval, as a fundamental task in the cross-modal field, aims to explore the relationship between visual and textual modalities. Recent methods address this task only by learning the conceptual and syntactical correspondences between cross-modal fragments, but these correspondences inevitably contain noise without considering external knowledge. To solve this issue, we propose a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> ommonsense-Guided <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> emantic and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b> elational <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> onsistencies (CSRC) for image-text retrieval that can simultaneously expand the semantics and relations to reduce the cross-modal differences under the assumption that the semantics and relations of the true image-text pair should be consistent between two modalities. Specifically, we first explore commonsense knowledge to expand the specific concepts for visual and textual graphs and optimize the semantic consistency by minimizing the differences in cross-modal semantic importance. Then, we extend the same relations for cross-modal concept pairs with semantic consistency, which serves to implement relational consistency. After that, we combine external commonsense knowledge with internal correlation to enhance concept representation and further optimize relational consistency by regularizing the importance differences between association-enhanced concepts. Extensive experimental results on two popular image-text retrieval datasets demonstrate the effectiveness of our proposed method.
Read moreMulti-View Attention Network for Visual Dialog
Visual dialog is a challenging vision-language task in which a series of questions visually grounded by a given image are answered. To resolve the visual dialog task, a high-level understanding of various multimodal inputs (e.g., question, dialog history, and image) is required. Specifically, it is necessary for an agent to (1) determine the semantic intent of question and (2) align question-relevant textual and visual contents among heterogeneous modality inputs. In this paper, we propose Multi-View Attention Network (MVAN), which leverages multiple views about heterogeneous inputs based on attention mechanisms. MVAN effectively captures the question-relevant information from the dialog history with two complementary modules (i.e., Topic Aggregation and Context Matching), and builds multimodal representations through sequential alignment processes (i.e., Modality Alignment). Experimental results on VisDial v1.0 dataset show the effectiveness of our proposed model, which outperforms previous state-of-the-art methods under both single model and ensemble settings.
Read moreMultimedia content analytics with modality transition
Multimedia integrating different modalities such as text, image, and video, provides users with much convenience in the digital era. Researchers have been establishing multimedia infrastructure over the recent decades, and nowadays multimedia content can be delivered to almost everyone and everywhere. With the fast development of the media world, the multimedia research community has focused attention on multimedia content analytics, which aims to recognise and represent semantic information from various data sources and content types. Vision and language are two representative content forms out of various multimedia formats. This dissertation investigates the interactions between vision and language modalities to enhance the comprehension ability of multimedia content analytics methods. The main challenges of multimedia content analytics come from the feature representations of visual and textual content and the intrinsic modality gap between them, as well as the time-consuming training process. From a visual content aspect, although a convolutional neural network based model can extract visual features that are effective for conventional computer vision tasks such as image classification, the learned representations have limitations when generalising to advanced visual comprehension tasks including image captioning and visual dialogue. On the other hand, from a language aspect, the language models learn word embeddings for textual content representation and generation. To generate high quality text, an image caption for instance, the model must be trained with high quality data. However, the quality of the training data cannot be guaranteed, and imperfect annotations will inevitably lead to output of subpar quality. In addition, modality transition and model training efficiency problems are worth investigating to further enhance the model usability. To address the aforementioned challenges, this dissertation concentrates on model effectiveness and efficiency. Firstly, the depth map and scene graph are exploited to enhance the visual representations derived from the image. Chapter 2 introduces the depth-aware attention model for image paragraph captioning. A depth map is estimated to augment visual cues for more accurate, logical and diverse paragraph generation. The object relationships are discovered for the visual dialogue model in Chapter 3. The objects and their interactions are extracted from the image to form a scene graph, and then the graph structure is preserved in a novel hierarchical graph convolutional network. The dialogue reasoning module can then benefit from the comprehensive visual features extracted via this process. Secondly, the effectiveness of the language model is investigated in Chapter 4. A number of annotation quality issues are identified in the image caption training data collected from an online crowd-sourcing platform. The human-consensus loss is proposed to allow the model to learn from training data that includes imperfect annotations by setting a high priority training focus on high quality annotations. Thirdly, the modality gap between vision and language is explicitly addressed with a modality transition module in Chapter 5. The proposed modality transition module ensures a smooth transition from visual features to semantic embeddings for more precise and context-aware caption generation. Lastly, Chapter 6 considers the training efficiency of the image captioning model. The training deficiency is addressed with a well-engineered attention mechanism which can be trained in parallel. The training time is, therefore, significantly reduced, whilst maintaining competitive model performance.
Read moreVisual Coreference Resolution in Visual Dialog Using Neural Module Networks
Visual dialog entails answering a series of questions grounded in an image, using dialog history as context. In addition to the challenges found in visual question answering (VQA), which can be seen as one-round dialog, visual dialog encompasses several more. We focus on one such problem called visual coreference resolution that involves determining which words, typically noun phrases and pronouns, co-refer to the same entity/object instance in an image. This is crucial, especially for pronouns (e.g., ‘it’), as the dialog agent must first link it to a previous coreference (e.g., ‘boat’), and only then can rely on the visual grounding of the coreference ‘boat’ to reason about the pronoun ‘it’. Prior work (in visual dialog) models visual coreference resolution either (a) implicitly via a memory network over history, or (b) at a coarse level for the entire question; and not explicitly at a phrase level of granularity. In this work, we propose a neural module network architecture for visual dialog by introducing two novel modules—Refer and Exclude—that perform explicit, grounded, coreference resolution at a finer word level. We demonstrate the effectiveness of our model on MNIST Dialog, a visually simple yet coreference-wise complex dataset, by achieving near perfect accuracy, and on VisDial, a large and challenging visual dialog dataset on real images, where our model outperforms other approaches, and is more interpretable, grounded, and consistent qualitatively.
Read moreReasoning with Multi-Structure Commonsense Knowledge in Visual Dialog
Visual Dialog requires an agent to engage in a conversation with humans grounded in an image. Many studies on Visual Dialog focus on the understanding of the dialog history or the content of an image, while a considerable amount of commonsense-required questions are ignored. Handling these scenarios depends on logical reasoning that requires commonsense priors. How to capture relevant commonsense knowledge complementary to the history and the image remains a key challenge. In this paper, we propose a novel model by Reasoning with Multi-structure Commonsense Knowledge (RMK). In our model, the external knowledge is represented with sentence-level facts and graph-level facts, to properly suit the scenario of the composite of dialog history and image. On top of these multi-structure representations, our model can capture relevant knowledge and incorporate them into the vision and semantic features, via graph-based interaction and transformer-based fusion. Experimental results and analysis on VisDial v1.0 and VisDialCK datasets show that our proposed model effectively outperforms comparative methods.
Read more