MSEE: Stochastic Cognitive Linguistic Behavior Models for Semantic Sensing

An efficient method to automatically recognize basic human actions is proposed to improve the communication between a human and a computer. Human actions are considered as patterns generated by complex non-linear dynamical models. A non-linear dynamical model is used to represent human actions. Gaussian process dynamical models are used to capture the spatial and temporal behaviors of actions. To make the process more efficient a 7-dimensional feature is extracted for each action. Although the extracted feature vector is compact compared to a high-dimensional temporal pattern, it can efficiently discriminate among different actions. The tests run on CMU MoCap database with SVM show promising results.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Depth sensors open up possibilities of dealing with the human action recognition problem by providing 3D human skeleton data and depth images of the scene. Analysis of human actions based on 3D skeleton data has become popular recently, due to its robustness and view-invariant representation. However, the skeleton alone is insufficient to distinguish actions which involve human-object interactions. In this paper, we propose a deep model which efficiently models human-object interactions and intra-class variations under viewpoint changes. First, a human body-part model is introduced to transfer the depth appearances of body-parts to a shared view-invariant space. Second, an end-to-end learning framework is proposed which is able to effectively combine the view-invariant body-part representation from skeletal and depth images, and learn the relations between the human body-parts and the environmental objects, the interactions between different human body-parts, and the temporal structure of human actions. We have evaluated the performance of our proposed model against 15 existing techniques on two large benchmark human action recognition datasets including NTU RGB+D and UWA3DII. The Experimental results show that our technique provides a significant improvement over state-of-the-art methods.

Recognition of human activities is an essential field in computer vision. The most human activity consists of the interaction between humans and objects. Many successful works have been done on human-object interaction (HOI) recognition and achieved acceptable results in recent years. Still, they are fully supervised and need to train labeled data for all HOIs. Due to the enormous space of human-object interactions, listing and providing the training data for all possible categories is costly and impractical. We propose an approach for scaling human-object interaction recognition in video data through the zero-shot learning technique to solve this problem. Our method recognizes a verb and an object from the video and makes an HOI class. Recognition of the verbs and objects instead of HOIs allows identifying a new combination of verbs and objects. So, a new HOI class can be identified, which is not seen by the recognizer system. We introduce a neural network architecture that can understand and represent the video data. The proposed system learns verbs and objects from available training data at the training phase and can identify the verb-object pairs in a video at test time. So, the system can identify the HOI class with different combinations of objects and verbs. Also, we propose to use lateral information for combining the verbs and the objects to make valid verb-object pairs. It helps to prevent the detection of rare and probably wrong HOIs. The lateral information comes from word embedding techniques. Furthermore, we propose a new feature aggregation method for aggregating extracted high-level features from video frames before feeding them to the classifier. We illustrate that this feature aggregation method is more effective for actions that include multiple subactions. We evaluated our system by recently introduced Charades challengeable dataset, which has lots of HOI categories in videos. We show that our proposed system can detect unseen HOI classes in addition to the acceptable recognition of seen types. Therefore, the number of classes identifiable by the system is greater than the number of classes used for training.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Two-person interaction recognition has become an area of growing interest in human action recognition. The graph convolutional network (GCN) using human skeleton data has been shown to be highly effective for action recognition. Most GCN-based methods focus on recognizing an individual person’s actions on the basis of an intra-body graph. However, many of these methods do not represent the relation between two bodies, making it difficult to accurately recognize human interaction. In this work, we propose multi-stream adaptive GCN using inter- and intra-body graphs (MAGCN-IIG) as a new method of human interaction recognition. To ensure highly accurate human interaction recognition, our method cooperatively utilizes two types of graphs: an inter-body graph and an intra-body graph. The inter-body graph, which is newly introduced in this paper, connects the inter-body joints between two people as well as intra-body connections. The adaptive GCN using the inter-body graph captures the relation of joints between two people, even different types of joints located far away from each other. Further, by implementing a multi-stream architecture, our method simultaneously captures both inter-body and intra-body relations in each of two units that represent the position and motion of people. Experiments on interaction recognition using two large-scale human action datasets, NTU RGB+D and NTU RGB+D 120, showed that our method recognized human interactions more accurately than state-of-the-art methods.

With the rise of computer vision, it is becoming more and more important to accurately recognize and evaluate human actions. However, the complexity, intraclass differences, and viewing angle changes of human actions significantly impact the accuracy of identification human actions. This paper proposed an action recognition and evaluation method based on skeleton information extraction. Briefly, We use Lightweight OpenPose to extract the key points of human skeleton and perform processing work, including video data cutting, deleting some key points, supplementing missing key points, filtering processing, feature extraction, etc. Through an in-depth exploration of related theoretical technologies, we proposed a model for recognition and evaluation of human table tennis actions with an ordinary camera. The support vector machine algorithm (SVM) classification model is used to identify table tennis actions in real-time. Then the dynamic time regularization (DTW) algorithm calculates the similarity of each human skeleton key point in the action sequence. The low-scoring bone key points are marked to evaluate the human table tennis action in real time. The results show that a recognition rate of more than 95% is achieved in the test set, which proves the method's effectiveness. In addition, we compared the results with previous work using inertial sensors for action recognition, which shows our method can preserve the same accuracy with a much lower cost of implementation.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

Regularized multi-task learning (MTL) algorithms have been exploited in the field of pattern recognition and computer vision gradually, which can fully excavate the relationships of different related tasks. Therefore, many dramatically favorable approaches based on regularized MTL have been proposed. In the past decades, although the promising results about human action recognition have been achieved, most of existing action recognition algorithms focus on action descriptors, single/multi-view and multi-modality action recognition, and few works are related with MTL, especial of lacking the systematic evaluation of existing MTL algorithms for human action recognition. Thus, in the paper, seven popular regularized MTL algorithms in which different actions are considered as different tasks, are systematically exploited on two public multi-view action datasets. In detail, dense trajectory features are firstly extracted for each view, and then the shared codebook are constructed for all views by k-means, and then each video is coded by the shared codebook. Moreover, according to different regularized MTL algorithms, all actions or part of actions are considered as related, and then these actions are set to different tasks in MTL. Further, the effectiveness of different number of training samples from different action views is also evaluated for MTL. Large scale experimental results show that: 1) Regularized MTL is very useful for action recognition which can dig the latent relationship among different actions; 2) Not of all human actions are related, if irrelative actions are put together in MTL, its performance will fall; 3) With the increase of the training samples from different views, the relationships about different actions can be fully exploited, and it promotes the accuracy improvement of action recognition.

Human action recognition is challenging, due to large temporal and spatial variations in actions performed by humans. These variations include significant nonlinear temporal stretching. In this paper, we propose an intuitively simple method to extract action templates from 3D human joint data that is insensitive to nonlinear stretching. The extracted action templates are used as the training instances of the actions to train multiple classifiers including a multi-class SVM classifier. Given an unknown action, we first extract and classify all its constituent atomic actions and then assign the action label via an equal voting scheme. We have tested the method on two public datasets that contain 3D human skeleton data. The experimental results show the proposed method can obtain a comparable or better performance than published state-of-the-art methods. Additional experiments also demonstrate the method works robustly on randomly stretched actions.

The recent evolution in sensor-based depth information has been developed a sounding scope to work for human activity recognition using depth image sequences. Activity recognition is having a key importance due to its advantages in several domain like surveillance systems at airport, patient monitoring system, care of elderly people, etc. This work proposed the objective and evaluations to recognise daily life human activities by spatiotemporal depth information. For this purpose, Kinect sensor is used to collect the data pertaining to a single activity performed by multiple person at a time. The spatiotemporal depth features are computed for activity recognition and support vector machine is used in classification phase. We have nine class of human actions in the database for RGB-D human activity recognition. This dataset is reconfigured from Cornell human activity and Berkeley multimodal human action databases. For multiple human action recognition, 91.38% accuracy is achieved on the synthetic dataset.

Scene image and human skeleton-based dual-stream human action recognition

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.