Action Recognition from One Example

In this paper we propose a novel method aiming at view-independent multi-view action recognition. Instead of combining the information provided by all the cameras forming the camera setup, for action representation and classification, we perform single-view action representation and classification to all the available videos depicting the person under consideration independently. Action representation involves a self organizing neural network training followed by fuzzy vector quantization. Action classification is performed by a feedforward neural network which is trained for view-invariant action recognition. Multiple action classification results combination based on Bayesian learning, in the recognition phase, results to high action recognition accuracy. The performance of the proposed action recognition method is evaluated on two publicly available databases, aiming at different application scenarios.

Most of the fast search motion estimation algorithms reduce the computational complexity of motion estimation (ME) greatly by checking only a few search points inside the search area. In this paper, we propose a new algorithm--multi-layer motion estimation (MME) which reduces the computational complexity of each distortion measure instead of reducing the number of search points. The conventional fast search motion estimation algorithms perform ME on the reference frame with full distortion measure; on the contrary, the MME performs ME on the layers with partial distortion measures to enhance the computational speed of ME. A layer is an image which is derived from the reference frame; each macro-pixel value in the layer represents the sum of the values of the corresponding pixels in the reference frame. A hierarchical quad-tree structure is employed in this paper to construct multiple layers from the reference frame. Experimental results on different video sequences show evidence that many motion vectors have been found similar both in the reference frame and the layers. The effectiveness of the proposed MME algorithm is compared with that of some state-of-the-art fast block matching algorithms with respect to speed and motion prediction quality. Experimental results on a wide variety of video sequences show that the proposed algorithm outperforms the other popular conventional fast search motion estimation algorithms computationally while maintaining the motion prediction quality very close to the full-search algorithm. Moreover, the proposed algorithm can achieve a maximum of 97.99 % speed-improvement rate against the fast full-search motion estimation algorithms which are based on hierarchical block matching process. The proposed MME performs the motion estimation on the layers by using three types of search patterns. The derivation of these search patterns exploits the characteristic of the center-biased motion vector distribution and that of less intensive block distortion measurement of the layers.

With the development of fast motion estimation (ME) algorithms towards video compression standard H.264, burden on integer-pel ME is slightly reduced. However the integer-pel ME is becoming computationally complex due to demand of higher and higher resolution videos. On the other hand, there is always trade-off between the speed and performance of these search algorithms. This work improvises the performance of content awareness enabled integer-pel ME algorithm for encoding the fast and slow motion video sequences of high definition (HD) resolution category viz. 1280×720 and 1920×1080. The algorithm proposes a novel notion of 'integrity index' with focus on increasing the PSNR and reducing the bit rate. It is implemented in the frame work of x265 version 1.7 video encoder. Motion independent ME algorithm is analysed quantitatively in terms of parameters based on rate distortion and ME time. The proposed algorithm is found to be performing better on BD-Rate and BD-PSNR parameters for the videos of both resolutions under consideration. Observed increase in motion estimation time is less than four seconds compared to hexagonal search algorithm of ME which is the benchmark in fast ME algorithm.

Action recognition on large categories of unconstrained videos taken from the web is a very challenging problem compared to datasets like KTH (6 actions), IXMAS (13 actions), and Weizmann (10 actions). Challenges like camera motion, different viewpoints, large interclass variations, cluttered background, occlusions, bad illumination conditions, and poor quality of web videos cause the majority of the state-of-the-art action recognition approaches to fail. Also, an increased number of categories and the inclusion of actions with high confusion add to the challenges. In this paper, we propose using the scene context information obtained from moving and stationary pixels in the key frames, in conjunction with motion features, to solve the action recognition problem on a large (50 actions) dataset with videos from the web. We perform a combination of early and late fusion on multiple features to handle the very large number of categories. We demonstrate that scene context is a very important feature to perform action recognition on very large datasets. The proposed method does not require any kind of video stabilization, person detection, or tracking and pruning of features. Our approach gives good performance on a large number of action categories; it has been tested on the UCF50 dataset with 50 action categories, which is an extension of the UCF YouTube Action (UCF11) dataset containing 11 action categories. We also tested our approach on the KTH and HMDB51 datasets for comparison.

The block motion estimation process is a very time consuming part in the H.264 video coding standard and thus hinders its real-time applications. Therefore, a new fast block motion estimation algorithm is proposed in this paper. In this algorithm, it first determines the current block as a quasi-stationary, slow motion, or fast motion block. Then, according to the different motion types, different search patterns and strategies are intelligently employed. Experimental results demonstrate that the proposed algorithm has much less execution time than the existing FS, UMHexagonS, and EPZS algorithms, with about the same accuracy.

This paper presents a method for recognizing human actions from a single query action video. We propose an action recognition scheme based on the ordinal measure of accumulated motion, which is robust to variations of appearances. To this end, we first define the accumulated motion image (AMI) using image differences. Then the AMI of the query action video is resized to a N×Nsubimage by intensity averaging and a rank matrix is generated by ordering the sample values in the sub-image. By computing the distances from the rank matrix of the query action video to the rank matrices of all local windows in the target video, local windows close to the query action are detected as candidates. To find the best match among the candidates, their energy histograms, which are obtained by projecting AMI values in horizontal and vertical directions, respectively, are compared with those of the query action video. The proposed method does not require any preprocessing task such as learning and segmentation. To justify the efficiency and robustness of our approach, the experiments are conducted on various datasets.

Few‐shot action recognition aims to recognise unseen actions given a few examples. Thus, this letter proposes a model named meta relation network (Meta RN) to address such problem. This model contains two parts: a MetaNet and a relation network. Relation network is utilised to extract video features and classify actions. A second‐order pooling followed by power normalization is used for feature enhancement, and target videos are finally classified by exploring nonlinear distance relations. The MetaNet module is designed to model different task distributions and generate task‐adaptive parameters for the embedding layer of the relation network in different tasks. Experimental results on two public action recognition datasets demonstrate that the network achieves higher accuracies than several state‐of‐the‐art approaches.

Action recognition makes significant contributions to sports video analysis, especially for athletes’ training evaluations. For sports video analysis, the action information is mainly conveyed by human body parts’ temporal movement, and each of the parts has a unique importance to the action representation. Aiming to involve this point in action recognition, we propose a part-attention spatio-temporal graph convolutional network (PSGCN) to exploit the dynamic spatio-temporal information in a sports video; it learns the importance of different parts to emphasize the contribution on the task of action recognition. Specifically, PSGCN first divides the human body into six parts and extracts their convolutional neural network (CNN) features, as well as concatenating the global feature of the whole frame; it then utilizes a cross-part and cross-frame graph building module to formulate the graph correlation of the parts from different frames. Inspired by the larger temporal variation of the same part containing more action information, we further propose a part-attention (PA) learning module to estimate the importance of each part, which can strengthen the graph correlation to support a PA graph. Finally, PSGCN conducts a graph convolutional network on the learned PA spatio-temporal graph with the learned part CNN features, which can obtain the action representation for the given sports video. In addition, the whole network is optimized by two losses of PA and action classification. To perform the superiority of PSGCN, we carry out extensive experiments of our model compared with several state-of-the-art methods on widely used action recognition datasets, especially for sports action. The results reflect the advantages of the proposed PSGCN on sports video analysis.

Action recognition in videos is a relevant and challenging task of automatic semantic video analysis. Most successful approaches exploit local space-time descriptors. These descriptors are usually carefully engineered in order to obtain feature invariance to photometric and geometric variations. The main drawback of space-time descriptors is high dimensionality and efficiency. In this paper we propose a novel descriptor based on 3D Zernike moments computed for space-time patches. Moments are by construction not redundant and therefore optimal for compactness. Given the hierarchical structure of our descriptor we propose a novel similarity procedure that exploits this structure comparing features as pyramids. The approach is tested on a public dataset and compared with state-of-the art descriptors.

Action recognition systems suffer significantly when dealing with rare or underepresented action classes. When training data are scarce, these systems show poor generalization and develop biased behavior that favors common actions. This study presents a novel GAN-based framework designed specifically to create realistic motion sequences for rare action classes in video datasets. The methodology combines temporal generative adversarial net-works with motion-aware discriminators. This combination produces high-quality synthetic action sequences that maintain both the visual appearance and timing patterns of real human movement. The approach uses a specialized architecture with three-dimensional convolutional layers, temporal attention mechanisms, and physical constraints. These components work together to ensure the generated motions look and feel realistic from a biomechanical perspective. The framework operates through a two-stage training process. First, a motion encoder analyzes existing rare action samples to extract key movement patterns. Then, a conditional GAN generates new motion sequences while keeping the es-sential characteristics that make each action unique. Testing on standard benchmarks (UCF-101, HMDB-51, and NTU RGB+D) shows substantial improvements in the rare ac-tion recognition accuracy. The synthetic data augmentation approach delivers an average performance boost of 23.7% for actions with fewer than 50 training samples and 31.2% for actions with fewer than 20 samples across all datasets. Complex actions show even more dramatic improvements, with medical procedures improving by 42.8% and special-ized sports movements by 38.4%. Quality assessment using Fréchet Video Distance (FVD) metrics and human perception studies confirms that generated motions are visually indis-tinguishable from real sequences. The synthetic movements maintain the characteristic timing patterns that define each action type. These findings demonstrate that AI-driven synthetic data generation can effectively solve class imbalance problems in action recogni-tion, leading to more robust and fair model performance across all action categories. Keywords: action recognition, synthetic data generation, generative adversarial networks, motion synthesis, class imbalance.

Zero-shot learning (ZSL) typically explores a shared semantic space in order to recognize novel categories in the absence of any labeled training data. However, the traditional ZSL methods always suffer from serious domain shift problem in human action recognition. This is because: 1) existing ZSL methods are specifically designed for object recognition from static images, which do not capture the temporal dynamics of video sequences, and poor performances are always generated if those methods are directly applied to zero-shot action recognition; 2) these methods always blindly project the target data into a shared space using a semantic mapping obtained by the source data without any adaptation, in which the underlying structures of target data are ignored; and 3) severe inter-class variations exist in various action categories. The traditional ZSL methods do not take relationships across different categories into consideration. In this paper, we propose a novel aligned dynamic-preserving embedding (ADPE) model for zero-shot action recognition in a transductive setting. In our model, an adaptive embedding of target videos is learned, exploring the distributions of both the source and target data. An aligned regularization is further proposed to couple the centers of target semantic representations with their corresponding label prototypes in order to preserve the relationships across different categories. Most significantly, during our embedding, the temporal dynamics of video sequences are simultaneously preserved via exploiting the temporal consistency of video sequences and capturing the temporal evolution of successive segments of actions. Our model can effectively overcome the domain shift problem in zero-shot action recognition. The experiments on Olympic sports, HMDB51, and UCF101 datasets demonstrate the effectiveness of our model.

With the rapid development of the power system and increasing demand for intelligence, substation operation training has received more attention. Action recognition is a monitoring and analysis system based on computer vision and artificial intelligence technology that can automatically identify and track personnel actions in video frames. The system accurately identifies abnormal behaviors such as illegal operations and provides real-time feedback to trainers or surveillance systems. The commonly adopted strategy for action recognition is to first extract human skeletons from videos and then recognize the skeleton sequences. Although graph convolutional networks (GCN)-based skeleton-based recognition methods have achieved impressive performance, they operate in spatial dimensions and cannot accurately describe the dependence between different time intervals in the temporal dimension. Additionally, existing methods typically handle the temporal and spatial dimensions separately, lacking effective communication between them. To address these issues, we propose a skeleton-based method that aggregates convolutional information of different scales in the time dimension to form a new scale dimension. We also introduce a space-time-scale attention module that enables effective communication and weight generation between the three dimensions for prediction. Our proposed method is validated on public datasets NTU60 and NTU120, with experimental results verifying its effectiveness. For substation operation training, we built a real-time recognition system based on our proposed method. We collected over 400 videos for evaluation, including 5 categories of actions, and achieved an accuracy of over 98%.

Image-guided liver surgery aims to enhance the precision of resection and ablation by providing fast localisation of tumours and adjacent complex vasculature to improve oncologic outcome. This Letter presents a novel end-to-end solution for fast stereo reconstruction and motion estimation that demonstrates high accuracy with phantom and clinical data. The authors’ computationally efficient coarse-to-fine (CTF) stereo approach facilitates liver imaging by accounting for low texture regions, enabling precise three-dimensional (3D) boundary recovery through the use of adaptive windows and utilising a robust 3D motion estimator to reject spurious data. To the best of their knowledge, theirs is the only adaptive CTF matching approach to reconstruction and motion estimation that registers time series of reconstructions to a single key frame for registration to a volumetric computed tomography scan. The system is evaluated empirically in controlled laboratory experiments with a liver phantom and motorised stages for precise quantitative evaluation. Additional evaluation is provided through testing with patient data during liver resection.

We propose a fast subpixel motion estimation algorithm for the H.264 advanced video coding (AVC) standard. The algorithm utilizes the correlation of the spatial interpolation effect on the full-pixel motion estimation best matches between different block sizes in order to reduce the computational cost of the overall motion-estimation process. Experimental results show that the proposed algorithm significantly reduces the CPU cycles in the various motion estimation schemes by up to 16% with similar rate-distortion performance when weighed up against the H.264/AVC standard.

With the rapid growth in the amount of video data, efficient video indexing and retrieval methods have become one of the most critical challenges in multimedia management. For this purpose, Content-Based Video Retrieval (CBVR) is nowadays an active area of research. In this article, a CBVR system providing similar videos from a large multimedia dataset based on query video has been proposed. This approach uses vector motion-based signatures to describe the visual content and uses machine learning techniques to extract key frames for rapid browsing and efficient video indexing. The proposed method has been implemented on both single machine and real-time distributed cluster to evaluate the real-time performance aspect, especially when the number and size of videos are large. Experiments were performed using various benchmark action and activity recognition datasets and the results reveal the effectiveness of the proposed method in both accuracy and processing time compared to previous studies.