Visual Prompting Through Image Mines

Abstract

Translate Article

Visual prompting aims to enhance the performance of vision-language models (VLMs), which, despite their remarkable capabilities, often struggle with dense, detailed images, leading to incorrect answers or hallucinations. We propose Visual Prompting Through Image Mines , a novel algorithm that leverages attention patterns from a base VLM to generate image crops for improved visual grounding. Specifically, we extract attention values from output text tokens in the LLaVA-8B model, overlay them onto image patches to create an attention graph, and apply a modified breadth-first search (BFS) to identify key regions as image crops. Using the SigLIP model, we refine these regions into Image Mines, retaining only the most relevant crops. Our approach supports both single-image and multi-image inference setups, delivering superior performance. It consistently outperforms existing visual prompting methods in nearly all scenarios.