• Home
  • Search
  • Semantic Embedding Guided Attention with Explicit Visual Feature Fusion for Video Captioning
  • Open Access IconOpen Access
  • Cite Icon13
  • https://doi.org/10.1145/3550276Copy DOI Icon

Semantic Embedding Guided Attention with Explicit Visual Feature Fusion for Video Captioning

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Video captioning, which bridges vision and language, is a fundamental yet challenging task in computer vision. To generate accurate and comprehensive sentences, both visual and semantic information is quite important. However, most existing methods simply concatenate different types of features and ignore the interactions between them. In addition, there is a large semantic gap between visual feature space and semantic embedding space, making the task very challenging. To address these issues, we propose a framework named semantic embedding guided attention with Explicit visual Feature Fusion for vidEo CapTioning, EFFECT for short, in which we design an explicit visual-feature fusion (EVF) scheme to capture the pairwise interactions between multiple visual modalities and fuse multimodal visual features of videos in an explicit way. Furthermore, we propose a novel attention mechanism called semantic embedding guided attention (SEGA ), which cooperates with the temporal attention to generate a joint attention map. Specifically, in SEGA, the semantic word embedding information is leveraged to guide the model to pay more attention to the most correlated visual features at each decoding stage. In this way, the semantic gap between visual and semantic space is alleviated to some extent. To evaluate the proposed model, we conduct extensive experiments on two widely used datasets, i.e., MSVD and MSR-VTT. The experimental results demonstrate that our approach achieves state-of-the-art results in terms of four evaluation metrics.

Similar Papers
  • Research Article
  • Citations13

Learning Multipart Attention Neural Network for Zero-Shot Classification

  • Dec 15, 2020
  • IEEE Transactions on Cognitive and Developmental Systems
  • Min Meng +2
  • Research Article
  • Citations12

Learning to Classify Fine-Grained Categories with Privileged Visual-Semantic Misalignment

  • Mar 01, 2017
  • IEEE Transactions on Big Data
  • Ke Chen +1
  • Research Article
  • Citations20

Bidirectional generative transductive zero-shot learning

  • Sep 12, 2020
  • Neural Computing and Applications
  • Xinpeng Li +5
  • Conference Article
  • Citations17

VS-Boost: Boosting Visual-Semantic Association for Generalized Zero-Shot Learning

  • Aug 01, 2023
  • Xiaofan Li +6
  • PDF
  • Research Article
  • Citations1

Explanatory Multi-Scale Adversarial Semantic Embedding Space Learning for Zero-Shot Recognition

  • Jan 01, 2022
  • Open Journal of Applied Sciences
  • Huiting Li
  • Research Article
  • Citations32

Quantitative Characterization of Semantic Gaps for Learning Complexity Estimation and Inference Model Selection

  • Oct 01, 2012
  • IEEE Transactions on Multimedia
  • Jianping Fan +4
  • Research Article
  • Citations28

Ventral & Dorsal Stream Theory based Zero-Shot Action Recognition

  • Mar 28, 2021
  • Pattern Recognition
  • Meng Xing +4
  • Research Article
  • Citations4

Learning discriminative visual semantic embedding for zero-shot recognition

  • Mar 14, 2023
  • Signal Processing: Image Communication
  • Yurui Xie +2
  • Research Article
  • Citations42

A Novel Perspective to Zero-Shot Learning: Towards an Alignment of Manifold Structures via Semantic Feature Expansion

  • Apr 03, 2020
  • IEEE Transactions on Multimedia
  • Jingcai Guo +1
  • Research Article
  • Citations47

Investigating the Bilateral Connections in Generative Zero-Shot Learning.

  • Feb 02, 2021
  • IEEE Transactions on Cybernetics
  • Jingjing Li +4
  • Book Chapter
  • Citations4

Zero-Shot Learning

  • Dec 07, 2018
  • Zhengming Ding +2
  • Research Article
  • Citations4

Modal-Dependent Retrieval Based on Mid-Level Semantic Enhancement Space

  • Jan 01, 2019
  • IEEE Access
  • Shunxin Zheng +3
  • Conference Article
  • Citations3

Generating Manifold-Aligned Semantic Feature for Zero-Shot Learning

  • Oct 01, 2018
  • Jidong Wang +3
  • Conference Article
  • Citations14

Generalised Zero-Shot Learning with Domain Classification in a Joint Semantic and Visual Space

  • Dec 01, 2019
  • Rafael Felix +3
  • Research Article
  • Citations48

A Variational Autoencoder with Deep Embedding Model for Generalized Zero-Shot Learning

  • Apr 03, 2020
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Peirong Ma +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.