VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidence

Visual understanding involves detecting objects in a scene and investigating rich semantic relationships between the objects, which is required for downstream visual reasoning tasks. The scene graph is widely used for structured scene representation, however, the performance of the scene graph generation for visual reasoning is limited due to challenges posed by imbalanced datasets and insufficient attention toward common sense knowledge infusion. Most of the existing approaches use statistical or language priors for knowledge infusion. Common Sense knowledge infusion using heterogeneous knowledge graphs can help in improving the accuracy, robustness, and generalizability of the scene graph generation and enable explainable higher level reasoning by providing rich and diverse background and factual knowledge about the concepts in visual scenes. In this article, we present the background and applications of the scene graph generation and the initial approaches and key challenges in common sense knowledge infusion using heterogeneous knowledge graphs for visual understanding and reasoning.

Computational complexity of linear constraints over the integers

The ability of an engineer to visualize and reason about geometric aspects of physical objects and processes is crucial in the success of engineering activities. The paper presents the development of a prototype visual reasoning system which aims at increasing engineering students’ visual reasoning capabilities. Specifically we take the missing view problem which requires students to construct any valid solid and a corresponding missing orthographic view when they are given two orthographic views. Sweeping operations, which are the reverse procedure of orthographic projection, are used to solve missing view problems by forming the boundary faces of a solid from the loops of orthographic views. This geometric operation forms a primary user interface of the system in visually reasoning about the construction of 3-D solid given the constraints by two orthographic views.

While image understanding on recognition-level has achieved remarkable advancements, reliable visual scene understanding requires comprehensive image understanding on recognition-level but also cognition-level, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge. In this paper, we propose a novel Cognitive Attention Network (CAN) for visual commonsense reasoning to achieve interpretable visual understanding. Specifically, we first introduce an image-text fusion module to fuse information from images and text collectively. Second, a novel inference module is designed to encode commonsense among image, query and response. Extensive experiments on large-scale Visual Commonsense Reasoning (VCR) benchmark dataset demonstrate the effectiveness of our approach. The implementation is publicly available at https://github.com/tanjatang/CAN.

Recent advances in text-only "slow-thinking" reasoning have prompted efforts to transfer this capability to vision-language models (VLMs), for training visual reasoning models (VRMs).However, such transfer faces critical challenges: Effective "slow thinking" in VRMs requires visual reflection, the ability to check the reasoning process based on visual information.Through quantitative analysis, we observe that current VRMs exhibit limited visual reflection, as their attention to visual information diminishes rapidly with longer generated responses.To address this challenge, we propose a new VRM Reflection-V 1 , which enhances visual reflection based on reasoning data construction for cold-start and reward design for reinforcement learning (RL).Firstly, we construct vision-centered reasoning data by leveraging an agent that interacts between VLMs and reasoning LLMs, enabling cold-start learning of visual reflection patterns.Secondly, a visual attention based reward model is employed during RL to encourage reasoning based on visual information.Therefore, Reflection-V demonstrates significant improvements across multiple visual reasoning benchmarks.Furthermore, Reflection-V maintains a stronger and more consistent reliance on visual information during visual reasoning, indicating effective enhancement in visual reflection capabilities.

This study introduces a novel benchmark dataset designed to evaluate the capabilities of Vision Language Models (VLMs) on tasks that combine visual reasoning with subject-specific background knowledge in the German language. In contrast to widely used English-language benchmarks that often rely on artificially difficult or decontextualized problems, this dataset draws from real middle school curricula across nine domains, including mathematics, history, and biology. The benchmark includes over 2,000 open-ended questions grounded in 486 images, ensuring that models must integrate visual interpretation with factual reasoning rather than rely on superficial textual cues. We evaluate thirteen state-of-the-art open-weight VLMs across multiple dimensions, including domain-specific accuracy and performance on adversarial crafted questions. Our findings reveal that even the strongest models achieve less than 45% overall accuracy, with particularly poor performance in music, mathematics, and adversarial settings. Furthermore, the results indicate significant discrepancies between success on popular benchmarks and real-world multimodal understanding. We conclude that middle school-level tasks offer a meaningful and underutilized avenue for stress-testing VLMs, especially in non-English contexts. The dataset and evaluation protocol serve as a rigorous testbed to better understand and improve the visual and linguistic reasoning capabilities of future AI systems.

Effectiveness and interpretability are two essential properties for trustworthy AI systems. Most recent studies in visual reasoning are dedicated to improving the accuracy of predicted answers, and less attention is paid to explaining the rationales behind the decisions. As a result, they commonly take advantage of spurious biases instead of actually reasoning on the visual-textual data, and have yet developed the capability to explain their decision making by considering key information from both modalities. This paper aims to close the gap from three distinct perspectives: first, we define a new type of multi-modal explanations that explain the decisions by progressively traversing the reasoning process and grounding keywords in the images. We develop a functional program to sequentially ex-ecute different reasoning steps and construct a new dataset with 1,040,830 multi-modal explanations. Second, we iden-tify the critical need to tightly couple important components across the visual and textual modalities for explaining the decisions, and propose a novel explanation generation method that explicitly models the pairwise correspon-dence between words and regions of interest. It improves the visual grounding capability by a considerable margin, resulting in enhanced interpretability and reasoning performance. Finally, with our new data and method, we perform extensive analyses to study the effectiveness of our explanation under different settings, including multi-task learning and transfer learning. Our code and data are available at https://github.com/szzexpoi/rex.

The research presented in this chapter is part of the R+D+I projects EDU2015-69731-R (Spanish Government/MinEco and ERDF) and GVPROMETEO2016-143 (Valencian Government).

Dynamic causal model diagnostic reasoning for online technical process supervision

A scheme for fuzzy temporal knowledge representation and reasoning is proposed. The scheme is based on the modification of the Petri nets called fuzzy time Petri nets and fuzzy temporal relations. The reasoning process uses modified Petri nets theory, possibility theory and a fuzzified version of Allen's (1983) temporal relations. The temporal linguistic expressions are modelled by primitives derived from the knowledge representation scheme. By combining and connecting these primitives the complex forms of situations from fuzzy temporal rich domains are modelled. An example of the use of the proposed scheme is given.

Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs).Unlike traditional image captioning or question answering tasks, rebus solving requires multi-modal abstraction, symbolic reasoning, and a grasp of cultural, phonetic and linguistic puns.In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles, ranging from simple pictographic substitutions to spatially-dependent cues ("head" over "heels").We analyze how different VLMs perform, and our findings reveal that while VLMs exhibit some surprising capabilities in decoding simple visual clues, they struggle significantly with tasks requiring abstract reasoning, lateral thinking, and understanding visual metaphors.

A review of fractional-order techniques applied to lithium-ion batteries, lead-acid batteries, and supercapacitors

Generating description for an image can be regard as visual understanding. It is across artificial intelligence, machine learning, natural language processing and many other areas. In this paper, we present a model that generates description for images based on RNN (recurrent neural network) with object attention and multi-feature of images. The deep recurrent neural networks have excellent performance in machine translation, so we use it to generate natural sentence description for images. The proposed method uses single CNN (convolution neural network) that is trained on ImageNet to extract image features. But we think it can not adequately contain the content in images, it may only focus on the object area of image. So we add scene information to image feature using CNN which is trained on Places205. Experiments show that model with multi-feature extracted by two CNNs perform better than which with a single feature. In addition, we make saliency weights on images to emphasize the salient objects in images. We evaluate our model on MSCOCO based on public metrics, and the results show that our model performs better than several state-of-the-art methods.

Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results underscore the benchmark's potential as a vital tool for advancing the field of multimodal modeling, providing a focused and challenging environment for enhancing model performance in visual and logical reasoning tasks.

In quest for an efficient representation schema for activity recognition in video, we employ techniques combining diagrammatic reasoning (DR) with qualitative spatial and temporal reasoning (QSTR). QSTR allows qualitative abstraction of spatio-temporal relations among objects of interest; and is often thwart by ambiguous conclusions. 'Diagrams' influence cognitive reasoning by externalizing mental context. Hence, QSTR over diagrams holds promise. We define 'diagrams' as explicit representation of objects of interest and their spatial information on a 2D grid. A sequence of 'key diagrams' is extracted. Inter diagrammatic reasoning operators combine 'key diagrams' to obtain spatio-temporal information. The qualitative spatial and temporal information thus obtained define short-term activity (STA). Several STAs combine to form long-term activities (LTA). Sequence of STAs as a feature vector is used for LTA recognition. We evaluate our approach over six LTAs from the CAVIAR dataset.