• Home
  • Search
  • Code-Driven LLM Agent for One-Shot Explanatory Visual Question Answering
  • https://doi.org/10.1145/3785327Copy DOI Icon

Code-Driven LLM Agent for One-Shot Explanatory Visual Question Answering

  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Code-driven Large Language Models (LLMs) integrate both natural and formal languages, enhancing reasoning, precision, and interaction with execution environments, which in turn augments the capabilities of intelligent agents. Recent advancements in code-driven LLMs have proven pivotal for visual tasks, such as Visual Question Answering (VQA), a critical task at the intersection of computer vision and natural language processing. Despite significant progress, interpretability remains a challenge for VQA models, leading to the emergence of multimodal explanations for VQA. In this paper, we propose OneCoLA (One-shot and training-free Code-driven LLM Agent), a novel framework for Multimodal Explanatory Visual Question Answering (MEVQA). OneCoLA enables LLMs to generate multimodal explanations for the VQA task by utilizing a one-shot prompt to convert input questions into Python programs that model the reasoning process. The framework supports the flexible integration of open-world tools, ensuring adaptability to different problem contexts. During program execution, OneCoLA captures and preserves key execution data to enhance the interpretability of the results. Additionally, through further one-shot prompting, the framework generates multimodal explanations by combining execution outcomes with relevant visual content, providing both textual and visual context. Experimental results compared with state-of-the-art methods demonstrate the effectiveness of OneCoLA in generating accurate and interpretable multimodal explanations without the need for extensive training.

Similar Papers
  • Conference Article

Grounded Multi-modal Conversation for Zero-shot Visual Question Answering

  • Oct 05, 2025
  • Mohammad Reza Zarei +2
  • Research Article

KGLMQA: enhancing medical visual question answering with knowledge graphs and LLMs

  • Mar 03, 2026
  • PeerJ Computer Science
  • Wenhu Wang +2
  • Research Article
  • Citations11

How Can IJDS Authors, Reviewers, and Editors Use (and Misuse) Generative AI?

  • Apr 01, 2023
  • INFORMS Journal on Data Science
  • Galit Shmueli +7
  • Research Article
  • Citations9

VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Ziyi Yin +7
  • Research Article
  • Citations54

Large language models for biomedicine: foundations, opportunities, challenges, and best practices.

  • Apr 24, 2024
  • Journal of the American Medical Informatics Association : JAMIA
  • Satya S Sahoo +8
  • Research Article
  • Citations7

Evaluating the performance of large language & visual-language models in cervical cytology screening

  • May 23, 2025
  • npj Precision Oncology
  • Qi Hong +15
  • Conference Article
  • Citations2

VQAR: Review on Information Retrieval Techniques based on Computer Vision and Natural Language Processing

  • Mar 01, 2019
  • Shivangi Modi +1
  • Research Article
  • Citations8

Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question Answering.

  • Aug 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhou Yu +4
  • Conference Article

CrossLM: A Data-Free Collaborative Fine-Tuning Framework for Large and Small Language Models

  • Jun 23, 2025
  • Yongheng Deng +5
  • Research Article
  • Citations2

LMT++: Adaptively Collaborating LLMs With Multi-Specialized Teachers for Continual VQA in Robotic Surgical Videos.

  • Jan 01, 2025
  • IEEE transactions on medical imaging
  • Yuyang Du +8
  • Research Article
  • Citations64

Computer Vision and Natural Language Processing

  • Dec 12, 2016
  • ACM Computing Surveys
  • Peratham Wiriyathammabhum +3
  • Research Article

Automating Software Release Notes with AI: A Comparative Study of Agent-Based Systems vs. LLM Fine-Tuning Approaches

  • Nov 17, 2025
  • International Scientific Journal of Engineering and Management
  • Abhishek Sharma
  • Research Article

#2924 Comparison of large language models and traditional natural language processing techniques in predicting arteriovenous fistula failure

  • May 23, 2024
  • Nephrology Dialysis Transplantation
  • Suman Lama +6
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
  • Research Article
  • Citations41

Prompting large language model with context and pre-answer for knowledge-based VQA

  • Mar 06, 2024
  • Pattern Recognition
  • Zhongjian Hu +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.