Multi-Modality Latent Interaction Network for Visual Question Answering

We propose a novel approach to identify the difficulty of visual questions for Visual Question Answering (VQA) without direct supervision or annotations to the difficulty. Prior works have considered the diversity of ground-truth answers of human annotators. In contrast, we analyze the difficulty of visual questions based on the behavior of multiple different VQA models. We propose to cluster the entropy values of the predicted answer distributions obtained by three different models: a baseline method that takes as input images and questions, and two variants that take as input images only and questions only. We use a simple k-means to cluster the visual questions of the VQA v2 validation set. Then we use state-of-the-art methods to determine the accuracy and the entropy of the answer distributions for each cluster. A benefit of the proposed method is that no annotation of the difficulty is required, because the accuracy of each cluster reflects the difficulty of visual questions that belong to it. Our approach can identify clusters of difficult visual questions that are not answered correctly by state-of-the-art methods. Detailed analysis on the VQA v2 dataset reveals that 1) all methods show poor performances on the most difficult cluster (about 10% accuracy), 2) as the cluster difficulty increases, the answers predicted by the different methods begin to differ, and 3) the values of cluster entropy are highly correlated with the cluster accuracy. We show that our approach has the advantage of being able to assess the difficulty of visual questions without ground-truth (i.e., the test set of VQA v2) by assigning them to one of the clusters. We expect that this can stimulate the development of novel directions of research and new algorithms.

Adversarial examples have been successfully generated for various image classification models. Recently, several methods have been proposed to generate adversarial examples for more sophisticated tasks such as image captioning and visual question answering (VQA). In this paper, we propose a targeted adversarial attack for VQA where the noise is added only to the background pixels of the image keeping the rest of the image unchanged. The experiments are done on two state-of-the-art VQA systems: End-to-End Neural Module Network (N2NMN) and Memory, Attention and Composition Network (MAC network) and three datasets: SHAPES, CLEVR, and VQA v2.0. We combine validation and test sets of SHAPES, and select 1000 image-question pairs from CLEVR validation set. For VQA v2.0, we select 500 image-question pairs from the validation set for experimentation. We study the proposed attack under two different settings: same-category and different-category; referring to whether or not the target adversarial answer lies in the same category as the original answer. For CLEVR, the proposed attack achieves 100% success rate for both the models under same-category setting and success rate of 22.3% for N2NMN and 73.9% for MAC network under different-category setting. For SHAPES, the proposed attack achieves success rate of 68.9% for N2NMN. The proposed attack also achieves high success rate for same-category setting in VQA v2.0. Furthermore, we give strong rationale behind the robustness of N2NMN to different-category attack.

We propose a model for free-form visual question answering (VQA) from human brain activity. The task of VQA is leading to an answer given an image and a question about the image. Given brain activity data measured by functional magnetic resonance imaging (fMRI) and a natural language question in terms of the viewed image, the proposed method can provide an accurate natural language answer with the VQA algorithm. Visual questions selectively approach various areas of an image such as objects and backgrounds. As a result, a more detailed understanding of the image and complex reasoning are typically needed than general image captioning models. In this paper, we propose a method of answering a given question about a viewed image from fMRI data based on the VQA algorithm. We estimate the relation between fMRI data and visual features extracted from viewed images. Based on the relationship, we convert fMRI data into visual features. Finally, the proposed method can answer to a visual question from fMRI data measured while subjects are viewing images. Experimental results show that the proposed method enables accurate answering for questions about viewed images.

The ability of Visual Question Answering (VQA) models to provide insightful responses to images based on their content holds the potential to completely transform machine interpretation and interaction. VQA systems face challenges due to noisy image data, making accurate question answering difficult. Several deep learning algorithms are used for predicting answers from images, but they don’t yield enough outcomes. To get over these problems, this work is suggested. In this manuscript, a visual question–answer model-based Transformer-Based Multi-Head Cross Attention Network with optimized Gold Rush Octave Convolution Network (T-MHCAN-OGRCNN) is proposed. To enhance VQA accuracy, three datasets are taken: VQA v2.0, OK-VQA, and FVQA, which include images with embedded questions and answers. These images often contain noise, necessitating pre-processing for improved results. Then, Task-Oriented Homogenized Automatic (TOHA) method for noise reduction is used. For word embedding, Multiscale Parallel MobileNetV3 (WE-MP-MobileNetV3) is employed. Following that, visual and textual features are extracted using Transformer-Based Multi-Head Cross Attention Network (T-MHCAN). The core VQA task is handled by a Fully Octave Convolution Network (OCNN), whose weight parameters are optimized using Gold Rush Optimizer (GRO). Therefore, this method incorporates cutting-edge strategies to improve precision and resilience of VQA systems. The suggested T-MHCAN-OGRCNN technique is implemented on Python platform. Three datasets were used to evaluate the effectiveness of the suggested T-MHCAN-OGRCNN, which achieves superior results than current techniques with 99.9% accuracy and 99.8% recall. This demonstrates approach’s exceptional effectiveness and room for growth in industry.

Visual question answering (VQA) is a task that machines should provide an accurate natural language answer given an image and a question about the image. Many studies have found that the current VQA methods are heavily driven by the surface correlation or statistical bias in the training data, and lack sufficient image grounding. To address this issue, we devise a novel end-to-end architecture that uses multitask learning to promote more sufficient image grounding and learn effective multimodality representations. The tasks consist of VQA and our proposed image cloze (IC) task requires machines to fill in the blanks accurately given an image and a textual description of the image. To ensure our model performs sufficient image grounding as much as possible, we propose a novel word-masking algorithm to develop the multimodal IC task based on the part-of-speech of words. Our model predicts the VQA answer and fills in the blanks after the multimodality representation learning that is shared by the two tasks. Experimental results show that our model achieves almost the equivalent, state-of-the-art, second-best performance on the VQA v2.0, VQA-changing priors (CP) v2, and grounded question answering (GQA) datasets, respectively, with fewer parameters and without additional data compared with baselines.

The discussion about what can Artificial Intelligence (AI) contribute to games has been running for a long time, however, recent advances in AI show promise of providing new kind of experiences for players and new tools for game developers. In contrast with the traditional Finite State Machine for interaction and response, we consider the scenario of Visual Question & Answering (VQA) - the automatic answering of a textual question about an image. VQA is a tool that can enrich possible answers by combining both visual and textual information. It is also trivial to extrapolate to a game setting without going away from the training domain. In this Work In Progress, we present two original prototypes designed to explore the potential of VQA in games and discuss preliminary findings originated through a Wizard of Oz (WOz) pilot study using VQA to investigate how people interact with such an AI algorithm.

Visual Question Answering (VQA) is the process of combining text and images to answer questions about images in natural language. A lot of the current methods use similar fusion techniques, which don't work well for more difficult queries because they can't capture both detailed semantic connections and broader contextual reasoning. To solve this problem, we present the Dual-Stream Semantic-Aware Fusion (DSSAF) model, which is meant to make it easier to combine and reason about multimodal features. DSSAF has two streams that work at the same time: a visual-semantic stream that connects object-level features with the question's context, and a global-context stream that brings in all the scene information. Cross-Modal Semantic Gating (CMSG) combines these streams in a smart way by changing how much each visual and text element contributes based on the question at hand. Also, a hierarchical reasoning block improves the inference by combining relational reasoning with multi-head cross-attention, which makes it possible to do deeper multi-step reasoning. Tests on VQA v2, GQA, and CLEVR show that DSSAF is more accurate than the best baselines, especially for questions about relationships and composition. Plus, qualitative visualizations show clear patterns of attention and gating, which makes the decision-making process more open. Keywords: Visual Question Answering, Multimodal Fusion, Cross-Modal Attention, Semantic Gating, Graph-Based Reasoning.

Language bias stands as a noteworthy concern in visual question answering (VQA), wherein models tend to rely on spurious correlations between questions and answers for prediction. This prevents the models from effectively generalizing, leading to a decrease in performance. In order to address this bias, we propose a novel modality fusion collaborative de-biasing algorithm (CoD). In our approach, bias is considered as the model's neglect of information from a particular modality during prediction. We employ a collaborative training approach to facilitate mutual modeling between different modalities, achieving efficient feature fusion and enabling the model to fully leverage multimodal knowledge for prediction. Our experiments on various datasets, including VQA-CP v2, VQA v2, and VQA-VS, using different validation strategies, demonstrate the effectiveness of our approach. Notably, employing a basic baseline model resulted in an accuracy of 60.14% on VQA-CP v2.

Image captioning and visual question answering are typical tasks that connect computer vision and natural language processing. Both of them need to effectively represent the visual content using computer vision methods and smoothly process the text sentence using natural language processing skills. The key problem of these two tasks is to infer the target result based on the interactive understanding of the word sequence and the image. Though they practically use similar algorithms, they are studied independently in the past few years. In this article, we attempt to exploit the mutual correlation between these two tasks. We propose the first VQA-improved image-captioning method that transfers the knowledge learned from the VQA corpora to the image-captioning task. A VQA model is first pretrained on image--question--answer instances. Then, the pretrained VQA model is used to extract VQA-grounded semantic representations according to selected free-form open-ended visual question--answer pairs. The VQA-grounded features are complementary to the visual features, because they interpret images from a different perspective. We incorporate the VQA model into the image-captioning model by adaptively fusing the VQA-grounded feature and the attended visual feature. We show that such simple VQA-improved image-captioning (VQA-IIC) models perform better than conventional image-captioning methods on large-scale public datasets.

One of the most overwhelming and exhilarating advancements in the field of intelligent systems has been the introduction of visual question answering (VQA), a new and exciting problem that makes use of natural language processing and computer vision for measuring the ability of a system for image understanding and generating inferences beyond object recognition, segmentation and image captioning. One of the primary goals of research in visual-based artificial intelligence is to design systems that can understand and reply to questions about visual data. The first part of the study very vividly presents numerous diversified as well as diagnostic datasets testing different visual reasoning abilities, while the second part of the study details a plethora of the recently developed approaches for VQA. Finally, a qualitative comparison of the diversity present in the various approaches has been done that can serve as an important benchmark for analysing and comparing the different characteristics and applications of these algorithms and datasets that can prove to be helpful to someone new to the field of VQA and its community.

In recent years, Visual Question Answering (VQA) has attracted increasing attention due to its requirement on cross-modal understanding and reasoning of vision and language. VQA is proposed to automatically answer natural language questions with reference to a given image. VQA is challenging, because the reasoning process on a visual domain needs a full understanding of the spatial relationship, semantic concepts, as well as the common sense for a real image. However, most existing approaches jointly embed the abstract low-level visual features and high-level question features to infer answers. These works have limited reasoning ability due to the lack of modeling of the rich spatial context of regions, high-level semantics of images, and knowledge across multiple sources. To solve the challenges, we propose multi-source multi-level attention networks for visual question answering that can benefit both spatial inferences by visual attention on context-aware region representation and reasoning by semantic attention on concepts as well as external knowledge. Indeed, we learn to reason on image representation by question-guided attention at different levels across multiple sources, including region and concept level representation from image source as well as sentence level representation from the external knowledge base. First, we encode region-based middle-level outputs from Convolutional Neural Networks (CNNs) into spatially embedded representation by a multi-directional two-dimensional recurrent neural network and, further, locate the answer-related regions by Multiple Layer Perceptron as visual attention. Second, we generate semantic concepts from high-level semantics in CNNs and select those question-related concepts as concept attention. Third, we query semantic knowledge from the general knowledge base by concepts and selected question-related knowledge as knowledge attention. Finally, we jointly optimize visual attention, concept attention, knowledge attention, and question embedding by a softmax classifier to infer the final answer. Extensive experiments show the proposed approach achieved significant improvement on two very challenging VQA datasets.

Visual question answering has recently been settled as a fundamental multi-modal reasoning task of artificial intelligence that allows users to get information about visual content by asking questions in natural language. In the cultural heritage domain, this task can contribute to assisting visitors in museums and cultural sites, thus increasing engagement. However, the development of visual question answering models for cultural heritage is prevented by the lack of suitable large-scale datasets. To meet this demand, we built a large-scale heterogeneous and multilingual (Italian and English) dataset for cultural heritage that comprises approximately 500K Italian cultural assets and 6.5M question-answer pairs. We propose a novel formulation of the task that requires reasoning over both the visual content and an associated natural language description, and present baselines for this task. Results show that the current state of the art is reasonably effective but still far from satisfactory; therefore, further research in this area is recommended. Nonetheless, we also present a holistic baseline to address visual and contextual questions and foster future research on the topic.

Recently, large multi-modal models (LMMs) have emerged with the capacity to perform vision tasks such as captioning and visual question answering (VQA) with unprecedented accuracy.Applications such as helping the blind or visually impaired have a critical need for precise answers.It is specially important for models to be well calibrated and be able to quantify their uncertainty in order to selectively decide when to answer and when to abstain or ask for clarifications.We perform the first indepth analysis of calibration methods and metrics for VQA with in-context learning LMMs.Studying VQA on two answerability benchmarks, we show that the likelihood score of visually grounded models is better calibrated than in their text-only counterparts for in-context learning, where sampling based methods are generally superior, but no clear winner arises.We propose AVG BLEU, a calibration score combining the benefits of both sampling and likelihood methods across modalities.

Recent advances in vision-language models have resulted in improved accuracy in visual question answering (VQA) tasks. However, their robustness remains limited when faced with out-of-distribution data containing unanswerable questions. In this study, we first construct a simple randomised VQA dataset, incorporating unanswerable questions from the VQA v2 dataset, to evaluate the robustness of a state-of-the-art VQA model. Our findings reveal that the model struggles to predict the “unknown” answer or provides inaccurate responses with high confidence scores for irrelevant questions. To address this issue without retraining the large backbone models, we propose Cross Modal Augmentation (CMA), a model-agnostic, test-time-only, multi-modal semantic augmentation technique. CMA generates multiple semantically-consistent but heterogeneous instances from the visual and textual inputs, which are then fed to the model, and the predictions are combined to achieve a more robust output. We demonstrate that implementing CMA enables the VQA model to provide more reliable answers in scenarios involving unanswerable questions, and show that the approach is generalisable across different categories of pre-trained vision language models.

A key solution to visual question answering (VQA) exists in how to fuse visual and language features extracted from an input image and question. We show that an attention mechanism that enables dense, bi-directional interactions between the two modalities contributes to boost accuracy of prediction of answers. Specifically, we present a simple architecture that is fully symmetric between visual and language representations, in which each question word attends on image regions and each image region attends on question words. It can be stacked to form a hierarchy for multi-step interactions between an image-question pair. We show through experiments that the proposed architecture achieves a new state-of-the-art on VQA and VQA 2.0 despite its small size. We also present qualitative evaluation, demonstrating how the proposed attention mechanism can generate reasonable attention maps on images and questions, which leads to the correct answer prediction.