A daily activity feature extraction approach based on time series of sensor events.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Cross-domain activity recognition via transfer learning

Smart Homes are generally considered the final solution for living problem, especially for the health care of the elderly and disabled, power saving, etc. Human activity recognition in smart homes is the key to achieving home automation, which enables the smart services to automatically run according to the human mind. Recent research has made a lot of progress in this field; however, most of them can only recognize default activities, which is probably not needed by smart homes services. In addition, low scalability makes such research infeasible to be used outside the laboratory. In this study, we unwrap this issue and propose a novel framework to not only recognize human activity but also predict it. The framework contains three stages: recognition after the activity, recognition in progress, and activity prediction in advance. Furthermore, using passive RFID tags, the hardware cost of our framework is sufficiently low to popularize the framework. In addition, the experimental result demonstrates that our framework can realize good performance in both activity recognition and prediction with high scalability.

Human activity recognition is an important area of ubiquitous computing. Most current researches in activity recognition mainly focus on simple activities, e.g., sitting, running, walking, and standing. Compared with simple activities, complex activities are more complicated with high-level semantics, e.g., working, commuting, and having a meal. This paper presents a hierarchical model to recognize complex activities as mixtures of simple activities and multiple actions. We generate the components of complex activities using a clustering algorithm, represent and recognize complex activities by applying a topic model on these components. It is a data-driven method that can retain effective information for representing and recognizing complex activities. In addition, acceleration and physiological signals are fused in classifier level to ensure the overall performance of complex activity recognition. The results of experiments show that our method has ability to represent and recognize complex activities effectively.

In recent years, activity recognition in smart homes is an active research area. It is an important problem of Human Computer Interaction (HCI), and has many applications in HCI, such as assistive living and healthcare. Recognition of users’ common behaviors allows an environment to provide personalized service. Unlike activity recognition in computer vision which uses cameras, it studies activities by embedded sensors in smart homes. In this paper, we propose a method to extract latent features from sensor data by Beta Process Hidden Markov Model (BP-HMM). The contributions of our method are twofold: 1, we extend BP-HMM by dependent Beta process, and integrate state constraints of sensors into the sampling process of BP-HMM. 2, we extract latent features automatically by our dependent BP-HMM, and train a structural support vector machine (SVM) by these features in a supervised way for activity recognition. To evaluate the proposed method, we performed experiments on the real-world smart home datasets. Our results suggest that extracting latent features from sensor data leads to good performance for activity recognition.

Action recognition in static image is challenging. The authors propose mutually incoherent pose bases which are implicit poselet co‐occurrences and are learned by dictionary training to describe body pose. Poselets in a pose basis are not constrained in space and quantity, thus pose basis can describe body pose more flexibly than k ‐poselet. In their method, body pose in an image is represented by a sparse linear combination of pose bases because pose in an action varies while each image only captures a snapshot from a single viewpoint. In dictionary training, the challenge is how to stabilise the sparse representation which is the input of Support Vector Machine (SVM) for action recognition, because the original pose signal is ambiguous while dictionary is an over complete matrix. Their solution is to add cumulative coherence as penalty in objective function and induce pose bases become mutually incoherent. They evaluate the method on two popular datasets and experiment results show the pose representation has encouraging performance in action recognition. Furthermore, they empirically exploit the complementary role of the local pose feature with deep convolutional neural network features from holistic image. Experiment results demonstrate aggressive performance improvement by concatenating the two features.

Activity recognition is an important component of many pervasive computing applications. Device-free activity recognition has the advantage that it does not have the privacy concern of using cameras and the subjects do not have to carry a device on them. Recently, it has been shown that channel state information (CSI) can be used for activity recognition in a device-free setting. With the proliferation of wireless devices, it is important to understand how radio frequency interference (RFI) can impact on pervasive computing applications. In this paper, we investigate the impact of RFI on device-free CSI-based location-oriented activity recognition. We conduct experiments in environments without and with RFI. We present data to show that RFI can have a significant impact on the CSI vectors. In the absence of RFI, different activities give rise to different CSI vectors that can be differentiated visually. However, in the presence of RFI, the CSI vectors become much noisier and activity recognition also becomes harder. Our extensive experiments shows that the performance of state-of-the-art classification methods may degrade significantly with RFI. We then propose a number of counter measures to mitigate the impact of RFI and improve the location-oriented activity recognition performance. Our evaluation shows the proposed method can improve up to 10% true detection rate in the presence of RFI. We also study the impact of bandwidth on activity recognition performance. We show that with a channel bandwidth of 20 MHz (which is used by WiFi), it is possible to achieve a good activity recognition accuracy when RFI is present.

It is challenging for an intelligent system to recognize the actions recorded in an RGB video due to the large amount of information and wide variations in the RGB video. On the other side, skeleton data focuses on the region of human body but lacks the interaction information with the background, which is complementary to the RGB data. Recently, some works focus on combining the RGB and skeleton data together to boost the performance of action recognition. However, the semantic information between joints is missing in existing works, which is important for action recognition. In this paper, we propose a novel semantic-augmented local decision aggregation network for action recognition. Specifically, we regard the area of body joints as the attention region to extract a local spatio-temporal feature for each body joint. In order to take advantage of the semantic information between joints, we propose a semantic information module, which jointly encodes the spatial and temporal index of body joints to enhance the representation ability of the local features. For better learning ability, instead of aggregating the local features, we first make decisions based on each individual local feature and then aggregate the local decisions for final recognition, which reflects the idea of resemble learning. Extensive experiments demonstrate the effectiveness of our proposed module which improves the performance of action recognition on three commonly used datasets.KeywordsAction recognitionAttentionLocal featuresDecision aggregationSemantic information

This study aims to design an epidemic prevention and control bracelet system that integrates convolutional neural network (CNN). This system can collect and process the user’s physiological index data in real time, especially in the remote physical education scene, and provide learners with immediate physiological index feedback and personalized adaptive training suggestions through accurate human action recognition (HAR) technology. One-dimensional acceleration signal is converted into two-dimensional image, and CNN’s powerful feature extraction and classification ability is used to effectively solve the problem that manual feature extraction is complex and nonlinear features are difficult to capture. By considering the joint action trajectory in the time window, a dynamic Recurrence Plot (RP) is constructed to capture the dynamic changes among joints. To input recursive graph data into CNN, it needs to be converted into image form. In the task of HAR, CNN can automatically learn useful features from images without manually designing features. It can not only effectively extract features from images, but also be directly used in classification tasks. Experimental results show that compared with other algorithms, the proposed RP + CNN model has the best performance in action recognition, with an accuracy of 96.89% and a F1 value of 86.76%. RP captures the dynamic patterns and periodic behaviors in time series by visualizing the repeated appearance of system states over time. The RP + CNN model is used to extract and classify human action features, which significantly improves the accuracy and efficiency of HAR. This innovative method not only simplifies the complex process of traditional manual feature extraction, but also enhances the system’s ability to identify nonlinear and complex action patterns, which provides strong technical support for remote physical education.

Video action recognition technology provides technical support for automated monitoring and early event warning, liberating human resources, preventing anomalies in advance, and handling events more promptly. However, the issue of user privacy leakage is also a growing concern. This paper discusses how to conduct action recognition while protecting privacy, proposing a generative adversarial network architecture that combines a multi-scale feature fusion generator with a spatiotemporal consistency discriminator. The network continuously enhances its capabilities through adversarial training strategies to protect facial privacy in videos. This network extracts features at different levels in the video through a multi-scale feature fusion mechanism so that the video after facial privacy protection still maintains a high degree of realism at the same time, to ensure that the accuracy of action recognition is not compromised, the feature enhancement module is designed to enhance action features and inhibit the privacy features significantly C3D is used as the action recognition model to accurately recognize a variety of actions in the video, such as running, jumping, falling, realizing the action analysis of video content. In this paper, the proposed method is evaluated in terms of privacy protection level and action recognition performance. Experiments are conducted on three datasets, LFW, HMDB51, and Hollywood2, and the results show that this framework effectively protects personal information while maintaining high action recognition accuracy.

In this work\footnote {This work was supported in part by the National Science Foundation under grant IIS-1212948.}, we present a method to represent a video with a sequence of words, and learn the temporal sequencing of such words as the key information for predicting and recognizing human actions. We leverage core concepts from the Natural Language Processing (NLP) literature used in sentence classification to solve the problems of action prediction and action recognition. Each frame is converted into a word that is represented as a vector using the Bag of Visual Words (BoW) encoding method. The words are then combined into a sentence to represent the video, as a sentence. The sequence of words in different actions are learned with a simple but effective Temporal Convolutional Neural Network (T-CNN) that captures the temporal sequencing of information in a video sentence. We demonstrate that a key characteristic of the proposed method is its low-latency, i.e. its ability to predict an action accurately with a partial sequence (sentence). Experiments on two datasets, \textit{UCF101} and \textit{HMDB51} show that the method on average reaches 95\% of its accuracy within half the video frames. Results, also demonstrate that our method achieves compatible state-of-the-art performance in action recognition (i.e. at the completion of the sentence) in addition to action prediction.

RGB and depth modalities contain more abundant and interactive information, and convolutional neural networks (ConvNets) based on multi-modal data have achieved successful progress in action recognition. Due to the limitation of a single stream, it is difficult to improve recognition performance by learning multi-modal interactive features. Inspired by the multi-stream learning mechanism and spatial-temporal information representation methods, we construct dynamic images by using the rank pooling method and design an interactive learning dual-ConvNet (ILD-ConvNet) with a multiplexer module to improve action recognition performance. Built on the rank pooling method, the constructed visual dynamic images can capture the spatial-temporal information from entire RGB videos. We extend this method to depth sequences to obtain more abundant multi-modal spatial-temporal information as the inputs of the ConvNets. In addition, we design a dual ILD-ConvNet with multiplexer modules to jointly learn the interactive features of two-stream from RGB and depth modalities. The proposed recognition framework has been tested on two benchmark multi-modal datasets—NTU RGB + D 120 and PKU-MMD. The proposed ILD-ConvNet with a temporal segmentation mechanism achieves an accuracy of 86.9% and 89.4% for Cross-Subject (C-Sub) and Cross-Setup (C-Set) on NTU RGB + D 120, 92.0% and 93.1% for Cross-Subject (C-Sub) and Cross-View (C-View) on PKU-MMD, which are comparable with the state of the art. The experimental results shown that our proposed ILD-ConvNet with a multiplexer module can extract interactive features from different modalities to enhance action recognition performance.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

Action recognition using spatial-optical data organization and sequential learning framework

In this paper, we propose an end-to-end deep learning framework that could extract the global spatial-temporal features from the videos based on the NetVladframework optimized with data augmentation. Given keyframes extracted from the original videos, we have presented a three-step action recognition framework: the first step of our framework is given by a data augmentation based on central crop, random crop and keyframe scaling. Then the second step is given by a local feature descriptor of each frame with Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation, which enables the network to capture spatial and temporal features simultaneously. The third step is given by aggregating global features for action recognition through the new generalized “Vector of Locally Aggregated Descriptors” (NetVLAD) layer optimized with a novel pooling strategy, avoiding the misjudgment caused by local features. The whole framework is trained and fine-tuned via an end-to-end way. It's demonstrated that the performance of our framework outperforms the state-of-the-art algorithms on UCF10l dataset. The competitive results clearly reveal an efficient action recognition of high accuracy (up to 91.25%) in fast time (close to 1.8s), which will significantly improve performance and application of video action recognition.