• Home
  • Search
  • Zero-Shot Cross-Lingual Knowledge Transfer in VQA via Multimodal Distillation
  • Cite Icon8
  • https://doi.org/10.1109/tcss.2024.3402270Copy DOI Icon

Zero-Shot Cross-Lingual Knowledge Transfer in VQA via Multimodal Distillation

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

As multilingual artificial intelligence systems proliferate, achieving robust cross-lingual understanding remains an open challenge. Recent works have made progress on visual question answering (VQA) models by pretraining on large English image-text datasets. However, there is a language gap as most models are English-centric. Existing attempts at multilingual VQA rely on machine translation or multilingual model pretraining, but cannot effectively transfer rich cross-modal knowledge from English models. In this work, we propose the cross-lingual multimodal knowledge transfer (CMKT) framework to efficiently extend English VQA models to non-English languages via knowledge distillation. Specifically, we introduce a code-mixed cross-lingual mask modeling (CCM) method to establish representations for new languages using small image-text data. We also design a multimodal knowledge distillation (MMKD) method to transfer modal understanding from English models by imitating their sequence processing. Experiments on the xGQA benchmark demonstrate that CMKT can effectively improve zero-shot learning and few-shot learning in non-English languages. Our method reduces the data and computation needed to train multilingual VQA models from scratch. The knowledge transfer paradigm enables non-English languages to inherit and generalize the intricate visual-semantic relationships learned from English. The results also show that the proposed method outperforms previous state-of-the-art methods in the zero-shot setting on the xGQA dataset.

Similar Papers
  • PDF
  • Conference Article
  • Citations2

Can Pre-training help VQA with Lexical Variations?

  • Jan 01, 2020
  • Shailza Jolly +1
  • Research Article
  • Citations1

Adversarial Sample Synthesis for Visual Question Answering

  • Nov 21, 2024
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Chuanhao Li +4
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
  • Research Article
  • Citations33

Causal Inference with Knowledge Distilling and Curriculum Learning for Unbiased VQA

  • Mar 04, 2022
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Yonghua Pan +3
  • PDF
  • Research Article
  • Citations19

Advancing surgical VQA with scene graph knowledge

  • May 23, 2024
  • International Journal of Computer Assisted Radiology and Surgery
  • Kun Yuan +5
  • Conference Article
  • Citations668

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

  • Jun 01, 2018
  • Aishwarya Agrawal +3
  • Conference Article

Estimation of Visual Contents from Human Brain Signals via VQA Based on Brain-Specific Attention

  • Jun 04, 2023
  • Ryo Shichida +4
  • Conference Article
  • Citations7

Learning to Ask Informative Sub-Questions for Visual Question Answering

  • Jun 01, 2022
  • Kohei Uehara +2
  • Conference Article
  • Citations47

Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

  • Jul 18, 2019
  • Yangyang Guo +5
  • Research Article

Visual Question Answering Model Using Transformer-Based Multi-Head Cross Attention Network with Optimized Gold Rush Octave Convolution Network

  • Oct 28, 2025
  • International Journal of Image and Graphics
  • J Jinu Sophia +2
  • Research Article
  • Citations20

A framework for visual question answering with the integration of scene-text using PHOCs and fisher vectors

  • Nov 07, 2021
  • Expert Systems with Applications
  • Himanshu Sharma +1
  • PDF
  • Research Article
  • Citations5

Visual Question Answering Models for Zero-Shot Pedestrian Attribute Recognition: A Comparative Study

  • Jun 28, 2024
  • SN Computer Science
  • Modesto Castrillón-Santana +5
  • Conference Article
  • Citations90

Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects

  • Jan 01, 2019
  • Gabriel Grand +1
  • Conference Article
  • Citations15

Feature Enhancement in Attention for Visual Question Answering

  • Jul 01, 2018
  • Yuetan Lin +3
  • Research Article
  • Citations30

Inverse Visual Question Answering: A New Benchmark and VQA Diagnosis Tool.

  • Nov 09, 2018
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Feng Liu +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.