Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Real-time identification of human activities in urban environments is increasingly becoming important in the context of public safety and national security. Distributed camera networks that provide multiple views of a scene are ideally suited for real-time action recognition. However, deployments of multi-camera based real-time action recognition systems have thus far been inhibited because of several practical issues and restrictive assumptions that are typically made such as the knowledge of a subjects orientation with respect to the cameras, the duration of each action and the conformation of a network deployment during the testing phase to that of a training deployment. In reality, action recognition involves classification of continuously streaming data from multiple views which consists of an interleaved sequence of various human actions. While there has been extensive research on machine learning techniques for action recognition from a single view, the issues arising in the fusion of data from multiple views for reliable action recognition have not received as much attention. In this thesis, I have developed a fusion framework for human action recognition using a multi-camera network that addresses these practical issues of unknown subject orientation, unknown view configurations, action interleaving and variable duration actions.;The proposed framework consists of two components: (1) a score-fusion technique that utilizes underlying view-specific supervised learning classifiers to classify an action within a given set of frames and (2) a sliding window technique that is used to parse a sequence of frames into multiple actions. The use of a score-fusion technique as opposed to a feature-level fusion of data from multiple views allows us to robustly classify actions even when camera configurations are arbitrary and different from training phase and at the same time reduces the required network bandwidth for data transmission permitting wireless deployments. Moreover, the proposed framework is independent of the underlying classifier that is used to generate scores for each action snippet and thus offers more flexibility compared to sequential approaches like Hidden Markov Models. The amount of training and parameterization is also significantly lower compared to HMM-based approaches. This Real-Time recognition system has been tested on 4 classifiers which are Linear Discriminant Analysis, Multinomial Naive Bayes, Logistic Regression and Support Vector Machines. Over 90% accuracy has been achieved by this system in Real-Time recognizing variable duration actions performed by the subject. The performance of the system is also shown to be robust to camera failures.

Activity recognition and classification is very useful filed. In this paper, we present the activity classification using a single tri-axial accelerometer of smartphone. Smartphones have a lot of sensors and a powerful performance, and many researchers study using smartphone sensors. Especially, utilization with the accelerometer is very large. The topic of the activity classification can be used as many parts, such as health-care part, medical part, and emergency part. We want to make the activity predictive model in everyday life using the activity classification. To make this model, user’s activity should be classified and recorded. In order to classify the daily activity, some elements should be considered. In this paper, we analyze to find the optimized environments for activity classification.

One of the essential tasks of healthcare and smart-living systems is to recognize the current activity of a particular user. Such activity recognition (AR) is demanding when only limited sensors are used, such as accelerometers. Given a small number of accelerometers, intelligent AR systems often use simple architectures, either general or specific for their AR. In this paper, a system for AR named TriLAR is presented. TriLAR has an AR-specific architecture consisting of three layers: (i) a bottom layer, where an arbitrary number of AR methods can be used to recognize the current activity; (ii) a middle layer, where the predictions from the bottom-layer methods are inputs for a hierarchical structure that combines domain knowledge and meta-classification; and (iii) a top layer, where a hidden Markov model is used to correct spurious transitions between the recognized activities from the middle layer. The middle layer has a hierarchical, three-level structure. First, a meta-classifier is used to make the initial separation between the most distinct activities. Second, domain knowledge in the form of rules is used to differentiate between the remaining activities, recognizing those of interest (i.e., static activities). Third, another meta-classifier deals with the remaining activities. In this way, each activity is recognized by the method best suited to it, leaving unrecognized activities to the next method. This architecture was tested on a dataset recorded using ten volunteers who acted out a complex, real-life scenario while wearing accelerometers placed on the chest, thigh, and ankle. The results show that TriLAR successfully recognized elementary activities using one or two sensors and significantly outperformed three standard, single-layer methods with all sensor placements.

Nowadays, the human behavior analysis by computer vision techniques has been an interesting issue for researchers. Automatic recognition of actions in video allows automation of many otherwise manually intensive tasks such as video surveillance. Video surveillance system especially for elderly care and their behavior analysis has an important role to take care of aged, impatient or bedridden persons. In this paper, we propose a high accuracy human action classification and recognition method using hidden Markov model classifier. In our approach, first, we use star skeleton feature extraction method to extract extremities of human body silhouette to produce feature vectors as inputs of hidden Markov model classifier. Then, hidden Markov model, which is learned and used in our proposed surveillance system, classifies the investigated behaviors and detects abnormal actions with high accuracy in comparison by other abnormal detection reported in previous works. The accuracy about 94% resulted from confusion matrix approve the efficiency of the proposed method when compared with its counterparts for abnormal action detection.

Encoding local motion information using spatio-temporal features is a common approach in action recognition methods. These features are based on the information content inside subregions extracted at locations of interest in a video. In this paper, we propose a conceptually different approach to video feature extraction. We adopt an entropy-based saliency framework and develop a method for estimating tube-like salient regions of flexible shape (i.e., tuboids). We suggest that the local shape of spatio-temporal subregions defined by changes in local information content can be used as a descriptor of the underlying motion. Our main goal in this paper is to introduce the concept of adaptive tuboid shapes as a local spatio-temporal descriptor. Our approach’s original idea is to use changes in local spatio-temporal information content to drive the tuboid’s shape deformation, and then use the tuboid’s shape as a local motion descriptor. Finally, we conduct a set of action recognition experiments on video sequences. Despite the relatively lower classification performance when compared to state-of-the-art action-recognition methods, our results indicate a good potential for the adaptive tuboid descriptor as an additional cue for action recognition algorithms.KeywordsAction RecognitionCamera ParameterHuman Action RecognitionInterest Point DetectorAxial CurveThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

- Action recognition is a popular research topic in the computer vision and machine learning domains. Although many action recognition methods have been proposed, only a few researchers have focused on cross-domain few-shot action recognition, which must often be performed in real security surveillance. Since the problems of action recognition, domain adaptation, and few-shot learning need to be simultaneously solved, the cross-domain few-shot action recognition task is a challenging problem. To solve these issues, in this work, we develop a novel end-to-end pairwise attentive adversarial spatiotemporal network (PASTN) to perform the cross-domain few-shot action recognition task, in which spatiotemporal information acquisition, few-shot learning, and video domain adaptation are realised in a unified framework. Specifically, the Resnet-50 network is selected as the backbone of the PASTN, and a 3D convolution block is embedded in the top layer of the 2D CNN (ResNet-50) to capture the spatiotemporal representations. Moreover, a novel attentive adversarial network architecture is designed to align the spatiotemporal dynamics actions with higher domain discrepancies. In addition, the pairwise margin discrimination loss is designed for the pairwise network architecture to improve the discrimination of the learned domain-invariant spatiotemporal feature. The results of extensive experiments performed on three public benchmarks of the cross-domain action recognition datasets, including SDAI Action I, SDAI Action II and UCF50-OlympicSport, demonstrate that the proposed PASTN can significantly outperform the state-of-the-art cross-domain action recognition methods in terms of both the accuracy and computational time. Even when only two labelled training samples per category are considered in the office1 scenario of the SDAI Action I dataset, the accuracy of the PASTN is improved by 6.1%, 10.9%, 16.8%, and 14% compared to that of the TA3N , TemporalPooling, I3D, and P3D methods, respectively.

Recently, activity recognition and classification (ARC) of human activity opens new research area in the field health care, security, and privacy of human society. Specifically, the promise of device-free activity recognition platform attracts researchers to develop platform to ensure the correct detection of activity recognition. The technologies, such as Wi-Fi, GSM, and radars, do not require installing cameras or wearable sensors for activity monitoring and recognition. Therefore, this device-free technology has gain popularity in health care and safety measurement systems. Traditional ARC systems depend on wearable sensors such as magic rings and vision technology such as a Microsoft Kinect. In the future, researchers are striving to reduce such devices and targeting a promising device-free sensing system. In this paper, a software-defined radio platform was designed for the detection of human activity. The extensive experiments were performed in the laboratory environment by using two Universal Software Radio Peripheral (USRP) to extract the wireless channel state information (WCSI). The 64-Fast Fourier Transform (FFT) point’s Orthogonal frequency division multiplexing (OFDM) signal was used to determine the WCSI. The design of the proposed system can be used for multiple applications due to scalability and flexibility of the software-defined hardware.

Action recognition in videos is currently in the focus of scientific research due to improvements made in automatic analysis of static images and greater availability of processing power. The paper provides an overview of the key models and methods for action recognition that comprise human models and methods based on estimation of joint trajectories, silhouettes and template matching and spatio-temporal local descriptors. To deal with compound actions and activities, action semantic models are proposed with help of expert knowledge. Since the action recognition task is domain dependent, the methods and models are built and tested on domain specific databases. The paper provides an overview and description of recent video datasets that were created for developing action recognition methods, with an emphasis on datasets with additional modalities such as depth images or accelerometer data.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Human gesture recognition is an area, which has been studied thoroughly in recent years,and close to100% recognition rates in restricted environments have been achieved, often either with single separated gestures in the input stream, or with computationally intensive systems. The results are unfortunately not as striking, when it comes to a continuous stream of gestures. In this paper we introduce a hierarchical system for gesture recognition for use in a gaming setting, with a continuous stream of data. Layer 1 is based on Nearest Neighbor Search and layer 2 uses Hidden Markov Models. The system uses features that are computed from Microsoft Kinect skeletons. We propose a new set of features, the relative angles of the limbs from Kinect's axes to use in NNS. The new features show a 10 percent point increase in precision when compared with features from previously published results. We also propose a way of attributing recognised gestures with a force attribute, for use in gaming. The recognition rate in layer 1 is 68.2%, with an even higher rate for simple gestures. Layer 2 reduces the noise and has aaverage recognition rate of 85.1%. When some simple constraints are added we reach a precision of 90.5% with a recall of 91.4%.

We present a system for integrating the neural networks' inference by using context and relation for complicated action recognition. In recent years, first person point of view which called as ego-centric video analysis draw a high attention to better understanding human activity and for being used to law enforcement, life logging and home automation. However, action recognition of ego-centric video is fundamental problem, and it is based on some complicating feature inference. In order to overcome these problems, we propose the context based inference for complicated action recognition. In realistic scene, people manipulate objects as a natural part of performing an activity, and these object manipulations are important part of the visual evidence that should be considered as context. Thus, we take account of such context for action recognition. Our system is consist of rule base architecture of bi-directional associative memory to use context of object-hand relationship for inference. We evaluate our method on benchmark first person video dataset, and empirical results illustrate the efficiency of our model.

Action Recognition in Assembly for Human-Robot-Cooperation using Hidden Markov Models

Face recognition based on multi-class mapping of Fisher scores

A novel feature generation scheme which combines multiclass mapping of Fisher scores and appearance based features for face recognition (FR) is proposed in this paper. Multi-class mapping of Fisher scores is based on partial derivative analysis of parameters of hidden Markov model (HMM), and appearance based features are obtained directed from face images. Linear discriminant analysis (LDA) is used to analyze the feature vectors generated under this scheme. Recognition performance improvement is observed over stand-alone HMM method as well as Fisherface method, which also uses appearance based feature vectors. Moreover, by reducing the number of models involved in the training and testing stages, the proposed feature generation scheme can maintain very high discriminative power at much lower computational complexity comparing to that of the traditional HMM based FR system. Experimental results are provided to demonstrate the viability of this scheme for face recognition.