3-D body joint-specific HMM-based approach for human activity recognition from stereo posture image sequence

Recognising human activity can be advantageous in a number of different scenarios including elder care, healthcare or for training purposes. It can be of direct use to support humans in doing different activities, but is still a challenge for systems to correctly classify the activity in a way that is valuable for the user, as they often times lack the robustness or simplicity for day-to-day use. In this paper an approach for human activity recognition based on object interactions is presented. The proposed system consists of a wireless sensor network, with each sensor node measuring the received signal strength indication (RSSI) to its neighbouring nodes. The accumulated RSSI data is then analyzed by a machine learning algorithm which tries to infer one of several cooked dishes from that data. Experimental studies demonstrate promising results and therefore potential for this technology for recognising human activity in the form of cooking, but its generalised approach makes it suitable for other environments, too.

Video and image analysis technologies have made human action recognition as an interesting field and used in many practical application systems such as visual surveillance systems, healthcare monitoring systems, 3D games and smart home. In this paper, we address the challenges of automatic tracking, detection and recognition of three-dimensional human pose-based actions from sequences of depth maps. Specifically, we have designed our proposed idea into a probabilistic framework: 1) dividing each action into meaningful ordered temporal segments, 2) using pixel-neighboring intensity difference approach to identify human shape region from the confused scenes, 3) introducing a robust spatiotemporal features to extract 3D joints information via local distance features and depth human shape information to get motion variation features and 4) finally these sets of features are merged together and their feature vectors are further reduced and discriminated to get optimal vectors for real-world applications. Next, the augmented features are enhanced and symbolized by Linde-Buzo-Gray clustering algorithm for better action recognition. With these symbols, each action hidden Markov model (HMM) is trained and tested for action recognition. The experimental results demonstrated our proposed approach on three challenging depth video datasets: IM-DepthActions, MSRAction3D and MSRDailyActivity3D, producing significantly improved results over the state-of-the-art methods: especially for those actions that are not easily discernible by the existing methods.

Activity and action detection, tracking and recognition are very demanding research area in computer vision and human computer interaction. In this paper, a video-based novel approach for human activity recognition is presented using robust hybrid features and embedded Hidden Markov Models. In the proposed HAR framework, depth maps are analyzed by temporal motion identification method to segment human silhouettes from noisy background and compute depth silhouette area for each activity to track human movements in a scene. Several representative features, including invariant, depth sequential silhouettes and spatiotemporal body joints features were fused together to explore gradient orientation change, intensity differentiation, temporal variation and local motion of specific body parts. Then, these features are processed by the dynamics of their respective class and learned, trained and recognized with specific embedded HMM having active feature values. Our experiments on two depth datasets demonstrate that the proposed features are efficient and robust over the state of the arts features for human activity recognition especially when there are similar postures of different activities.

In this paper, we present a “key segment” mining approach for human action recognition. Our model is able to locate discriminative segments for action samples via multiple instance learning. Moreover, we propose a dynamic pooling approach to automatically find the optimal length of segment for each action sample. In addition, an effective feature is proposed for action recognition with 3D skeleton joints. It can effectively capture informative motion and shape cues of skeletons, and leads to a compact and discriminative representation. The experimental results validate the effectiveness of the proposed human action recognition method on two benchmark datasets (i.e., MSR Action3D and UCF-Kinect). Moreover, our method demonstrates superior accuracy than previous methods of using only skeleton data on MSR Action3D, and achieves the state-of-the-art performance on UCF-Kinect.Keywordsaction recognitionmultiple instance learningskeleton featuredynamic pooling

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

Human action recognition has been an active research area in recent years. However, building a robust human action recognition system still remains a challenging task due to the large variations in action classes, varying human appearances, illumination changes, camera motion, occlusions and background clutter. Most previous work focus on the goal of improving recognition rates. This paper describes a computationally fast votingbased approach for human action recognition, in which the action in the video sequence is recognized based on the support of the local spatio-temporal features. The proposed technique requires no parameter tuning and can produce recognition rates that are comparable to those in recent published literature. Moreover, the technique can localize the single human action in the video sequence without much additional computation. Recognition results on the KTH and Weizmann action dataset are presented.

With the rapid increase in digital technology, most research areas are involved in human activity recognition, which can help to analyze the activities of patients. A novel approach for human activity recognition in egocentric video has been invoked in this research article. Generally, only the objects are identified, but the actions are not recognized. With this motivation and new trends, this paper presents an efficient technique to recognize the activities. In our approach, first the various activity dataset is trained, and the feature vector values are stored for various activities, which are applied to the testing inputs. Here, we use a filtering technique, i.e., a median filter followed by a segmentation method using watershed and feature extraction, such as a Histogram of Oriented Gradient (HOG), Color and GiST and a combination of all Features. Features are reduced using a genetic algorithm, and classification is done using Support Vector Machine (SVM) and a Random Forest classifier. The experimental results demonstrate that the Random Forest classifier outperformed the SVM classifier.

A multi-resolution fusion approach for human activity recognition from video data in tiny edge devices

Recognizing human activities is one of the crucial capabilities that a robot needs to have to be useful around people. Although modern robots are equipped with various types of sensors, human activity recognition (HAR) still remains a challenging problem, particularly in the presence of noisy sensor data. In this work, we introduce a multimodal graphical attention-based HAR approach, called Multi-GAT, which hierarchically learns complementary multimodal features. We develop a multimodal mixture-of-experts model to disentangle and extract salient modality-specific features that enable feature interactions. Additionally, we introduce a novel message-passing based graphical attention approach to capture cross-modal relation for extracting complementary multimodal features. The experimental results on two multimodal human activity datasets suggest that Multi-GAT outperformed state-of-the-art HAR algorithms across all datasets and metrics tested. Finally, the experimental results with noisy sensor data indicate that Multi-GAT consistently outperforms all the evaluated baselines. The robust performance suggests that Multi-GAT can enable seamless human-robot collaboration in noisy human environments.

This paper presents a new approach for human action recognition around sleeping with the human body parts locations and the positional relationship between human and sleeping environment. Body parts are estimated from the depth image obtained by a time-of-flight (TOF) sensor using oriented 3D normal vector. Issues in action recognition of sleeping situation are the demand of availability in darkness, and hiding of the human body by duvets. Therefore, the extraction of image features is difficult since color and edge features are obscured by covers. Thus, first in our method, positions of four parts of the body (head, torso, thigh, and lower leg) are estimated by using the shape model of bodily surface constructed by oriented 3D normal vector. This shape model can represent the surface shape of rough body, and is effective in robust posture estimation of the body hidden with duvets. Then, action descriptor is extracted from the position of each body part. The descriptor includes temporal variation of each part of the body and spatial vector of position of the parts and the bed. Furthermore, this paper proposes hierarchical action classes and classifiers to improve the indistinct action classification. Classifiers are composed of two layers, and recognize human action by using the action descriptor. First layer focuses on spatial descriptor and classifies action roughly. Second layer focuses on temporal descriptor and classifies action finely. This approach achieves a robust recognition of obscured human by using the posture information and the hierarchical action recognition.

Human action recognition (HAR)in videos is a challenging task in computer vision. Conventional methods are prone to explore the spatiotemporal or optical representations for video actions. However, optical representation might be inefficient in some real-life situations, such as object occlusion and dim light. To address this issue, this paper presents a novel approach for human action recognition by jointly exploiting video and Wi-Fi clues. We leverage the fact that Wi-Fi signals carry discriminative information of human actions, which is robust to optical limitations. To validate this innovative thought, we conceive a practical framework for HAR and setup a dataset containing both video clips and Wi-Fi Channel State Information of human actions. The 3D convolutional neural network was used to extract the video features and the statistical algorithms were used to extract radio features. A classical linear support vector machine is employed as the classifier after the video and radio feature fusion. Comprehensive experiments on this dataset achieved desirable results with the maximum improvement in accuracy by 10%. This demonstrates our promising findings: with the aid of Wi-Fi Channel State Information, the performance of the video action recognition methods can be improved significantly, even under the optical limitation.

In this paper we introduce a general probabilistic graphical model for human everyday activity recognition. The proposed model is a discriminative graphical model with hidden variables for modeling body pose and sequential order of them. We use a unified framework for prediction task that is faster and more efficient than structured support vector machine and hidden conditional random fields. We have trained and tested the model on RGB-D videos and the result was comparable to the state of the art.

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

ABSTRACTIn the present work, we have proposed an efficient approach for human action recognition (HAR) from silhouette image sequence in videos. The efficiency of the approach lies in feature extraction and action classification. The proposed approach includes scale-shift normalization and distorted silhouette removal for the extraction of newly introduced spatiotemporal features coined as active region energy feature (AREF), and trajectory analysis. On the other hand, classification is done using hierarchical structure. An active region is the changing region in two consecutive silhouettes to accomplish the action. The AREF is estimated using active region energy image (AREI), which embraces the energies of active regions. The higher values in the AREI signify the more activeness (changing) of that region across the silhouette sequences; i.e. the region is used more (active) to complete the action. The silhouette normalization technique makes the feature extraction more robust and scale invariant. Also, the proposed approach uses a low-dimensional feature vector, which makes the whole procedure effective regarding cost in terms of timing requirement. The experimental results on publicly available Weizmann and MuHVAi data-sets clearly validate the efficiency of the proposed technique on that of the related research work regarding accuracy in the human action detection.