Human Action Recognition in Videos Using Kinematic Features and Multiple Instance Learning

Every year, new Convolutional Neural Network (CNN) architectures appear to deal with different problems in the activity of image and video recognition. These architectures usually work along the ImageNet dataset for looking for the best performance of the CNNs without taking into account the video task where they are used. This can represent a problem if the task is Human Action Recognition (HAR) in video, since the CNN architectures are pre-trained with an image dataset that can practically contain any object, while HAR problem requires consecutive frames of people doing actions. To prove the idea that using CNNs pre-trained on an image dataset does not always achieve the best performance on a video dataset and that, therefore, it is worth comparing the performance of different CNNs under similar circumstances for the HAR problem, this work proposes an analysis between eight different CNN architectures. Each one of the CNN was exclusively trained with RGB images, which were extracted from the frames of the different classes of videos of HMDB51 dataset. To make the classification of an activity in video, we average the predictions taking into account the successes. We also made some ensembles with the best performance CNNs to measure the improvement in accuracy. Our results suggest that Xception is a strong baseline model that could be used by the community to make their comparisons of their proposals more robust.

Human action recognition in videos based on spatiotemporal features and bag-of-poses

Human action recognition in videos is a desired field in computer vision applications since it can be applied in human computer interaction, surveillance monitors, robot vision, etc. Two approaches of features are investigated in this chapter. First approach is a contour-based type. Four features are investigated in this approach such as Cartesian Coordinate Features (CCF), Fourier Descriptors Features (FDF), Centroid-Distance Features (CDF), and Chord-Length Features (CLF). The second approach is a silhouette-based type. Three features are investigated in this approach such as Histogram of Oriented Gradients (HOG), Histogram of Oriented Optical Flow (HOOF), and Structural Similarity Index Measure (SSIM) features. All these features are simple to compute, efficient to classify, and fast to calculate. Therefore, these features demonstrate a promising field for human action recognition. Moreover, the classification is achieved using two classifiers: K-Nearest-Neighbor (KNN) and Support Vector Machine (SVM). The experimental results demonstrated that these features have a promising potential and useful for the human action recognition in videos.

To handle with the limitation of bag-of-features (BoF) model which ignores spatial and temporal relationships of local features in human action recognition in video, a Local Spatiotemporal Coding (LSC) is proposed. Rather than the exiting methods only uses the feature appearance information for coding, LSC encodes feature appearance and spatiotemporal positions information simultaneously with vector quantization (VQ). It can directly models the spatiotemporal relationships of local features in space time volume (STV). In implement, the local features are projected into sub-space-time-volume (sub-STV), and encoded with LSC. In addition a multi-level LSC is also provided. Then a group of sub-STV descriptors obtained from videos with multi-level LSC and Avg-pooling are used for action video classification. A sparse representation based classification method is adopted to classify action videos upon these sub-STV descriptors. The experimental results on KTH, Weizmann, and UCF sports datasets show that our method achieves better performance than the previous local spatiotemporal features based human action recognition methods.

Human action recognition in surveillance has become a hot topic in computer vision. In this paper, we develope a new method to recognize human action using motion information in video. Video sequence is compressed along time axis into a Motion Impression Image (MII), which is combined with two types of impression images from different views. One is a Period Impression Image (PII) by exploring the characteristics of the motion frequency. The other is an Optical Flow Impression Image (OFII) obtained from the analysis of motion mode. The proposed MII is a compact and time-invariant representation. Furthermore, it is simple and efficient to implement. After quantizing the combined MIIs, we feed them into a spatial pyramid matching kernel (SPMK) based classifier to recognize various human actions. At last, experiments on a known benchmark dataset demonstrate the better performance of the proposed approach against the state-of-the-art algorithms.

Human action recognition in video has found widespread applications in many fields. However, this task is still facing many challenges due to the existence of intra-class diversity and inter-class overlaps among different action categories. The key trick of action recognition lies in the extraction of more comprehensive features to cover the action, as well as a compact and discriminative video encoding representation. Based on this observation, in this paper we propose a hybrid feature descriptor, which combines both static descriptor and motional descriptor to cover more action information inside video clips. We also adopt the usage of VLAD encoding method to encapsulate more structural information within the distribution of feature vectors. The recognition effects of our framework are evaluated on three benchmark datasets: KTH, Weizmann, and YouTube. The experimental results demonstrate that the hybrid descriptor, facilitated with VLAD encoding method, outperforms traditional descriptors by a large margin.

Human action recognition in videos is an important task with a broad range of applications. In this study, we improve the performance of recurrent attention convolutional neural network (RACNN) by proposing a novel model, “attention-again”. We consider the nature of video frames as sequences, which will cause the change of regions of interest in the frame, thus we cannot use an attention mechanism similar to that in images. “Attention-again” model is a variant from traditional attention model for recognizing human activities and is embedded in two long short-term memory (LSTM) layers. Different from hierarchal LSTM which change the LSTM structure to combine the hidden states from two LSTM layers, our proposals introduce “attention-again” model to avoid the change of LSTM structure. Furthermore, this model not only learns the relations in each frame, but also obtains the relations among all frames, and these relations instruct the next learning stage. Therefore, our proposed model outperform the baseline and is superior to methods with the same experimental conditions on three benchmark datasets: UCF-11, HMDB-51 and UCF-101. To understand how the model works, we also visualize the region of interest in the frame.

Human action recognition in video is often approached by means of sequential probabilistic models as they offer a natural match to the temporal dimension of the actions. However, effective estimation of the models' parameters is critical if one wants to achieve significant recognition accuracy. Parameter estimation is typically performed over a set of training data by maximizing objective functions such as the data likelihood or the conditional likelihood. However, such functions are nonconvex in nature and subject to local maxima. This problem is major since any solution algorithm (expectation-maximization, gradient ascent, variational methods and others) requires an arbitrary initialization and can only find a corresponding local maximum. Exhaustive search is otherwise impossible since the number of local maxima is unknown. While no theoretical solutions are available for this problem, the only practicable mollification is to repeat training with different initializations until satisfactory cross-validation accuracy is attained. Such a process is overall empirical and highly time-consuming. In this paper, we propose two methods for one-off initialization of hidden Markov models achieving interesting tradeoffs between accuracy and training time. Experiments over three challenging human action video datasets (Weizmann, MuHAVi and Hollywood Human Actions) and with various feature sets measured from the frames (STIP descriptors, projection histograms, notable contour points) prove that the proposed one-off initializations are capable of achieving accuracy above the average of repeated random initializations and comparable to the best. In addition, the methods proposed are not restricted solely to human action recognition as they suit time series classification as a general problem.

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

Action recognition in still images is closely related to various other computer vision tasks such as pose estimation, object recognition, image retrieval, video action recognition and frame tagging in videos. This problem is focused on recognizing a person's action or behavior using a single frame. Unlike action recognition in videos - a relatively very well established area of research where spatio-temporal features are used, these are not available for still images, making the problem more challenging. In the present work only actions that involve objects are considered. A complex action is broken down into components based on semantics. The importance of each of these components in action recognition is systematically studied.

In order to improve the accuracy of human action recognition in video and the computational efficiency of large data sets, an action recognition algorithm based on multiple features and modified deep learning model is proposed. First, the deep network pre-training process is used to learn and optimize the RBM parameters, and the deep belief nets (DBN) model is constructed through deep learning. Then, human 13 joint points and critical points of optical flow are automatically extracted by DBN model. Second, these more abstract and more effective human motion features are combined to represent human actions. Ultimately, the entire DBN network structure is fine-tuned by support vector machine (SVM) algorithm to classify human actions. We demonstrate that human 13 joint points and critical points of optical flow are two very effective human action characterizations, our proposed approach greatly reduces the required samples, and shortens the training time of the samples, can efficiently process large data sets and can effectively recognize novel actions. We performed experiments on the KTH data set, Weizmann data set, the ballet data set and UCF101 data set to evaluate the proposed method, the experiment results show that the average recognition accuracy is over 98%, which validates its effectiveness, and show that our results are stable, reliable, and significantly better than the results of two state-of-the-art approaches on four different data sets. So, it lays a good theoretical foundation for practical applications.

Action recognition is of great importance in understanding human motion from video. It is an important topic in computer vision due to its many applications such as video surveillance, human–machine interaction and video retrieval. One key problem is to automatically recognize low-level actions and high-level activities of interest. This paper proposes a way to cope with low-level actions by combining information of human body joints to aid action recognition. This is achieved by using high-level features computed by a convolutional neural network which was pre-trained on Imagenet, with articulated body joints as low-level features. These features are then used to feed a Long Short-Term Memory network to learn the temporal dependencies of an action. For pose prediction, we focus on articulated relations between body joints. We employ a series of residual auto-encoders to produce multiple predictions which are then combined to provide a likelihood map of body joints. In the network topology, features are processed across all scales which capture the various spatial relationships associated with the body. Repeated bottom-up and top-down processing with intermediate supervision of each auto-encoder network is applied. We demonstrate state-of-the-art results on the popular FLIC, LSP and UCF Sports datasets.

Non-negative matrix completion for action detection

Landmark-based human action recognition in videos is a challenging task in computer vision. One key step is to design a generic approach that generates discriminative features for the spatial structure and temporal dynamics. To this end, we regard the evolving landmark data as a high-dimensional path and apply path signature techniques to provide an expressive, robust, non-linear, and interpretable representation for the sequential events. We do not extract signature features from the raw path, rather we propose path disintegrations and path transformations as preprocessing steps. Path disintegrations turn a high-dimensional path linearly into a collection of lower-dimensional paths; some of these paths are in pose space while others are defined over a multi-scale collection of temporal intervals. Path transformations decorate the paths with additional coordinates in standard ways to allow the truncated signatures of transformed paths to expose additional features. For spatial representation, we apply the non-linear signature transform to vectorize the paths that arise out of pose disintegration, and for temporal representation, we apply it again to describe this evolving vectorization. Finally, all the features are joined together to constitute the input vector of a linear single-hidden-layer fully-connected network for classification. Experimental results on four diverse datasets demonstrated that the proposed feature set with only a linear shallow network is effective and achieves comparable state-of-the-art results to the advanced deep networks, and meanwhile, is capable of interpretation.

The majority of methods for recognizing human actions are based on single-view video or multi-camera data. In this paper, we propose a novel multi-surface video analysis strategy. The video can be expressed as three-surface motion feature (3SMF) and spatio-temporal interest feature. 3SMF is extracted from the motion history image in three different video surfaces: horizontal–vertical, horizontal- and vertical-time surface. In contrast to several previous studies, the prior probability is estimated by 3SMF rather than using a uniform distribution. Finally, we model the relationship score between each video and action as a probability inference to bridge the feature descriptors and action categories. We demonstrate our methods by comparing them to several state-of-the-arts action recognition benchmarks.