• Home
  • Search
  • Effective action recognition with fully supervised and self-supervised methods
  • Open Access IconOpen Access
  • https://doi.org/10.32657/10356/152741Copy DOI Icon

Effective action recognition with fully supervised and self-supervised methods

  • Jan 1, 2021
  • Haozhi Cao
Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Action recognition in videos has attracted interest in computer vision and machine learning communities thanks to its applications such as surveillance and smart homes. In addition to spatial information in individual frames, videos contain temporal information across the temporal dimension. Therefore, effective spatio-temporal representation is the key to accurate action recognition in videos. Previous works have proposed various fully-supervised and self-supervised methods for video representation learning. For fully-supervised methods, most of them utilize convolution neural networks (CNNs) to extract spatial representation while temporal representation is usually modelled by pixel-wise correlations. However, it is inefficient to extract correlations between all pixels since some of them may relate to in-salient area (e.g. backgrounds or environments). On the other hand, self-supervised methods are proposed to leverage more accessible unlabled data on the Internet and transfer the extracted representation for different downstream tasks. The core of self-supervised methods is to design a pretext task where supervision signal is automatically generated based on characteristics of unlabeled data. Although self-supervised methods avoid the annotation of labeled data, compared to fully-supervised methods, there is room for performance improvement of self-supervised methods. In this paper, we address the above research gap with two novel deep learning methods, to advance fully-supervised and self-supervised methods, respectively. For fully-supervised learning, we propose a novel Key Point Shift Embedding Module (KPSEM) to adaptively extract channel-wise key point shifts across video frames without key point annotation for temporal feature extraction. Key points are adaptively extracted as feature points with maximum feature values at split regions, while key point shifts are the spatial displacements of corresponding key points. The key point shifts are encoded as the overall temporal features via linear embedding layers in a multi-set manner. To advance self-supervised learning, we propose a novel self-supervised learning method, called Video Incoherence Detection (VID), that leverages incoherence detection for spatio-temporal feature extraction. It roots from the observation that visual systems of human beings can easily identify video incoherence based on their comprehensive understanding of videos. Specifically, the training sample, denoted as the incoherent clip, is constructed by multiple sub-clips hierarchically sampled from the same raw video with various lengths of incoherence between each other. The network is trained to learn high-level representation by predicting the relative location and length of incoherence given the incoherent clip as input. Additionally, intra-video contrastive learning is introduced to maximize the mutual information between different incoherent clips from the same raw video. Our experiments show that both KPSEM and VID achieve state-of-the-art performance on action recognition with fully-supervised and self-supervised learning, respectively. Thorough ablation studies are also conducted to justify the performance of both proposed methods.

Similar Papers
  • Conference Article
  • Citations33

Understanding action recognition in still images

  • Jun 01, 2020
  • Deeptha Girish +2
  • Book Chapter
  • Citations6

Audio and Video Feature Fusion for Activity Recognition in Unconstrained Videos

  • Jan 01, 2006
  • José Lopes +1
  • Research Article
  • Citations10

GA-STIP: Action Recognition in Multi-Channel Videos With Geometric Algebra Based Spatio-Temporal Interest Points

  • Jan 01, 2018
  • IEEE Access
  • Rui Wang +4
  • Conference Article
  • Citations3

Human action recognition in surveillance video of a computer laboratory

  • Aug 01, 2016
  • Abdul-Lateef Yussiff +2
  • Book Chapter
  • Citations4

Action Recognition in Untrimmed Videos with Composite Self-attention Two-Stream Framework

  • Jan 01, 2020
  • Dong Cao +2
  • Dissertation

Robust and efficient deep learning methods for vision-based action recognition

  • Jan 01, 2021
  • Yuecong Xu
  • Research Article
  • Citations14

Benchmarking Self-Supervised Contrastive Learning Methods for Image-Based Plant Phenotyping

  • Jan 01, 2023
  • Plant Phenomics
  • Franklin C Ogidi +2
  • Conference Article

Accelerated action recognition and segmentation in videos

  • Apr 07, 2021
  • Emna Ghodhbani +2
  • PDF
  • Research Article
  • Citations17

Hierarchical dynamic depth projected difference images–based action recognition in videos with convolutional neural networks

  • Jan 01, 2019
  • International Journal of Advanced Robotic Systems
  • Hanbo Wu +2
  • Book Chapter
  • Citations2

Exploiting Sub-region Deep Features for Specific Action Recognition in Combat Sports Video

  • Jan 01, 2018
  • Yongqiang Kong +4
  • Book Chapter

Action Recognition in Haze Using an Efficient Fusion of Spatial and Temporal Features

  • Jan 01, 2021
  • Sri Girinadh Tanneru +1
  • Conference Article
  • Citations28

DarkLight Networks for Action Recognition in the Dark

  • Jun 01, 2021
  • Rui Chen +4
  • Research Article

Study on Various Self-supervised Video Representation Learning Methods

  • Aug 31, 2022
  • Moving Image & Technology (MINT)
  • Soohyun Park +1
  • PDF
  • Research Article
  • Citations6

Enhancing Zero-Shot Action Recognition in Videos by Combining GANs with Text and Images

  • May 05, 2023
  • SN Computer Science
  • Kaiqiang Huang +2
  • Research Article
  • Citations80

Learning Transferable Self-Attentive Representations for Action Recognition in Untrimmed Videos with Weak Supervision

  • Jul 17, 2019
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Xiao-Yu Zhang +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.