• Home
  • Search
  • Zero-shot Cross-modal Retrieval by Assembling AutoEncoder and Generative Adversarial Network
  • Cite Icon39
  • https://doi.org/10.1145/3424341Copy DOI Icon

Zero-shot Cross-modal Retrieval by Assembling AutoEncoder and Generative Adversarial Network

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Conventional cross-modal retrieval models mainly assume the same scope of the classes for both the training set and the testing set. This assumption limits their extensibility on zero-shot cross-modal retrieval (ZS-CMR), where the testing set consists of unseen classes that are disjoint with seen classes in the training set. The ZS-CMR task is more challenging due to the heterogeneous distributions of different modalities and the semantic inconsistency between seen and unseen classes. A few of recently proposed approaches are inspired by zero-shot learning to estimate the distribution underlying multimodal data by generative models and make the knowledge transfer from seen classes to unseen classes by leveraging class embeddings. However, directly borrowing the idea from zero-shot learning (ZSL) is not fully adaptive to the retrieval task, since the core of the retrieval task is learning the common space. To address the above issues, we propose a novel approach named Assembling AutoEncoder and Generative Adversarial Network (AAEGAN), which combines the strength of AutoEncoder (AE) and Generative Adversarial Network (GAN), to jointly incorporate common latent space learning, knowledge transfer, and feature synthesis for ZS-CMR. Besides, instead of utilizing class embeddings as common space, the AAEGAN approach maps all multimodal data into a learned latent space with the distribution alignment via three coupled AEs. We empirically show the remarkable improvement for ZS-CMR task and establish the state-of-the-art or competitive performance on four image-text retrieval datasets.

Similar Papers
  • Research Article
  • Citations38

Learning Cross-Aligned Latent Embeddings for Zero-Shot Cross-Modal Retrieval

  • Apr 03, 2020
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Kaiyi Lin +4
  • Research Article
  • Citations222

Zero-VAE-GAN: Generating Unseen Features for Generalized and Transductive Zero-Shot Learning.

  • Jan 01, 2020
  • IEEE Transactions on Image Processing
  • Rui Gao +7
  • Research Article
  • Citations72

Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval

  • May 18, 2021
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Shengsheng Qian +4
  • Research Article
  • Citations7

Defect identification method for ultrasonic inspection of pipeline welds based on cross-modal zero-shot learning

  • Nov 02, 2023
  • Measurement Science and Technology
  • Yu Zeyu +3
  • Research Article
  • Citations20

Bidirectional generative transductive zero-shot learning

  • Sep 12, 2020
  • Neural Computing and Applications
  • Xinpeng Li +5
  • Research Article
  • Citations35

MFF: Multi-modal feature fusion for zero-shot learning

  • Sep 09, 2022
  • Neurocomputing
  • Weipeng Cao +4
  • PDF
  • Research Article
  • Citations6

Enhancing Zero-Shot Action Recognition in Videos by Combining GANs with Text and Images

  • May 05, 2023
  • SN Computer Science
  • Kaiqiang Huang +2
  • Conference Article
  • Citations22

Correlated Features Synthesis and Alignment for Zero-shot Cross-modal Retrieval

  • Jul 25, 2020
  • Xing Xu +4
  • Conference Article
  • Citations17

Towards Zero-Shot Learning with Fewer Seen Class Examples

  • Jan 01, 2021
  • Vinay Kumar Verma +4
  • Conference Article
  • Citations308

Generative Dual Adversarial Network for Generalized Zero-Shot Learning

  • Jun 01, 2019
  • He Huang +3
  • Research Article
  • Citations10

Dual Projective Zero-Shot Learning Using Text Descriptions

  • Jan 05, 2023
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Yunbo Rao +4
  • Research Article
  • Citations215

Ternary Adversarial Networks With Self-Supervision for Zero-Shot Cross-Modal Retrieval.

  • Jul 24, 2019
  • IEEE Transactions on Cybernetics
  • Xing Xu +5
  • Conference Article
  • Citations70

OntoZSL: Ontology-enhanced Zero-shot Learning

  • Apr 19, 2021
  • Yuxia Geng +7
  • Research Article
  • Citations21

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

  • Feb 01, 2023
  • IEEE Transactions on Circuits and Systems for Video Technology
  • Lei Liao +2
  • PDF
  • Research Article
  • Citations1

GT-GAN: A General Transductive Zero-Shot Learning Method Based on GAN

  • Jan 01, 2020
  • IEEE Access
  • Junhao Dong +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.