A Multi-resolution Action Recognition Algorithm Using Wavelet Domain Features

Human action recognition (HAR)in videos is a challenging task in computer vision. Conventional methods are prone to explore the spatiotemporal or optical representations for video actions. However, optical representation might be inefficient in some real-life situations, such as object occlusion and dim light. To address this issue, this paper presents a novel approach for human action recognition by jointly exploiting video and Wi-Fi clues. We leverage the fact that Wi-Fi signals carry discriminative information of human actions, which is robust to optical limitations. To validate this innovative thought, we conceive a practical framework for HAR and setup a dataset containing both video clips and Wi-Fi Channel State Information of human actions. The 3D convolutional neural network was used to extract the video features and the statistical algorithms were used to extract radio features. A classical linear support vector machine is employed as the classifier after the video and radio feature fusion. Comprehensive experiments on this dataset achieved desirable results with the maximum improvement in accuracy by 10%. This demonstrates our promising findings: with the aid of Wi-Fi Channel State Information, the performance of the video action recognition methods can be improved significantly, even under the optical limitation.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

Recognising human activity can be advantageous in a number of different scenarios including elder care, healthcare or for training purposes. It can be of direct use to support humans in doing different activities, but is still a challenge for systems to correctly classify the activity in a way that is valuable for the user, as they often times lack the robustness or simplicity for day-to-day use. In this paper an approach for human activity recognition based on object interactions is presented. The proposed system consists of a wireless sensor network, with each sensor node measuring the received signal strength indication (RSSI) to its neighbouring nodes. The accumulated RSSI data is then analyzed by a machine learning algorithm which tries to infer one of several cooked dishes from that data. Experimental studies demonstrate promising results and therefore potential for this technology for recognising human activity in the form of cooking, but its generalised approach makes it suitable for other environments, too.

In this paper, a stereo camera-based novel approach for Human Activity Recognition (HAR) is presented using robust 3-D human body joint features and joint-specific Hidden Markov Models (HMMs). At f...

This paper presents a new approach for human action recognition around sleeping with the human body parts locations and the positional relationship between human and sleeping environment. Body parts are estimated from the depth image obtained by a time-of-flight (TOF) sensor using oriented 3D normal vector. Issues in action recognition of sleeping situation are the demand of availability in darkness, and hiding of the human body by duvets. Therefore, the extraction of image features is difficult since color and edge features are obscured by covers. Thus, first in our method, positions of four parts of the body (head, torso, thigh, and lower leg) are estimated by using the shape model of bodily surface constructed by oriented 3D normal vector. This shape model can represent the surface shape of rough body, and is effective in robust posture estimation of the body hidden with duvets. Then, action descriptor is extracted from the position of each body part. The descriptor includes temporal variation of each part of the body and spatial vector of position of the parts and the bed. Furthermore, this paper proposes hierarchical action classes and classifiers to improve the indistinct action classification. Classifiers are composed of two layers, and recognize human action by using the action descriptor. First layer focuses on spatial descriptor and classifies action roughly. Second layer focuses on temporal descriptor and classifies action finely. This approach achieves a robust recognition of obscured human by using the posture information and the hierarchical action recognition.

In this article, we propose a novel multimodal data analytics scheme for human activity recognition. Traditional data analysis schemes for activity recognition using heterogeneous sensor network setups for eHealth application scenarios are usually a heuristic process, involving underlying domain knowledge. Relying on such explicit knowledge is problematic when aiming to create automatic, unsupervised or semi-supervised monitoring and tracking of different activities, and detection of abnormal events. Experiments on a publicly available OPPORTUNITY activity recognition database from UCI machine learning repository demonstrates the potential of our approach to address next generation unsupervised automatic classification and detection approaches for remote activity recognition for novel, eHealth application scenarios, such as monitoring and tracking of elderly, disabled and those with special needs.

Human action recognition has been an active research area in recent years. However, building a robust human action recognition system still remains a challenging task due to the large variations in action classes, varying human appearances, illumination changes, camera motion, occlusions and background clutter. Most previous work focus on the goal of improving recognition rates. This paper describes a computationally fast votingbased approach for human action recognition, in which the action in the video sequence is recognized based on the support of the local spatio-temporal features. The proposed technique requires no parameter tuning and can produce recognition rates that are comparable to those in recent published literature. Moreover, the technique can localize the single human action in the video sequence without much additional computation. Recognition results on the KTH and Weizmann action dataset are presented.

Motion is the most informative cue for human action recognition. Regions with high motion saliency indicate where actions occur and contain visual information that is most relevant to actions. In this paper, we propose a novel approach for human action recognition based on oriented motion salient regions (OMSRs). Firstly, we apply a bank of 3D Gabor filters and an opponent inhibition operator to detect OMSRs of videos, each of which corresponds to a specific motion direction. Then, a new low-level feature, named as oriented motion salient descriptor (OMSD), is proposed to describe the obtained OMSRs through the statistics of the texture in the regions. Next, we utilize the obtained OMSDs to explore the oriented characteristics of action classes and generate a set of class-specific oriented attributes (CSOAs) for each class. These CSOAs provide a compact and discriminative middle-level representation for human actions. Finally, an SVM classifier is utilized for human action classification and a new compatibility function is devised for measuring how well a given action matches to the CSOAs of a certain class. We test the proposed approach on four public datasets and the experimental results validate the effectiveness of our approach.KeywordsVideo SequenceAction ClassGabor FilterHuman Action RecognitionCompatibility FunctionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

How to deal with multi-modality data from different types of devices is a challenging issue for accurate recognition of human activities in a smart environment. In this paper, we propose a multimodal fusion enabled ensemble approach. Firstly, useful features collected from Bluetooth beacons, binary sensors, and smart floor are extracted and presented by fuzzy logic based-method with variable-size temporal windows. Secondly, a group of support vector machine classifiers are used to perform the classification task. Finally, a weighted ensemble method is used to obtain the final prediction. Especially, by applying the geometric framework, we are able to obtain the optimal weights for the ensemble. The proposed approach is evaluated on the UJAmI dataset. The experimental results demonstrate the efficacy and robustness of the proposed method.

In this paper, we present a “key segment” mining approach for human action recognition. Our model is able to locate discriminative segments for action samples via multiple instance learning. Moreover, we propose a dynamic pooling approach to automatically find the optimal length of segment for each action sample. In addition, an effective feature is proposed for action recognition with 3D skeleton joints. It can effectively capture informative motion and shape cues of skeletons, and leads to a compact and discriminative representation. The experimental results validate the effectiveness of the proposed human action recognition method on two benchmark datasets (i.e., MSR Action3D and UCF-Kinect). Moreover, our method demonstrates superior accuracy than previous methods of using only skeleton data on MSR Action3D, and achieves the state-of-the-art performance on UCF-Kinect.Keywordsaction recognitionmultiple instance learningskeleton featuredynamic pooling

In this paper we introduce a general probabilistic graphical model for human everyday activity recognition. The proposed model is a discriminative graphical model with hidden variables for modeling body pose and sequential order of them. We use a unified framework for prediction task that is faster and more efficient than structured support vector machine and hidden conditional random fields. We have trained and tested the model on RGB-D videos and the result was comparable to the state of the art.

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

Activity and action detection, tracking and recognition are very demanding research area in computer vision and human computer interaction. In this paper, a video-based novel approach for human activity recognition is presented using robust hybrid features and embedded Hidden Markov Models. In the proposed HAR framework, depth maps are analyzed by temporal motion identification method to segment human silhouettes from noisy background and compute depth silhouette area for each activity to track human movements in a scene. Several representative features, including invariant, depth sequential silhouettes and spatiotemporal body joints features were fused together to explore gradient orientation change, intensity differentiation, temporal variation and local motion of specific body parts. Then, these features are processed by the dynamics of their respective class and learned, trained and recognized with specific embedded HMM having active feature values. Our experiments on two depth datasets demonstrate that the proposed features are efficient and robust over the state of the arts features for human activity recognition especially when there are similar postures of different activities.

Video and image analysis technologies have made human action recognition as an interesting field and used in many practical application systems such as visual surveillance systems, healthcare monitoring systems, 3D games and smart home. In this paper, we address the challenges of automatic tracking, detection and recognition of three-dimensional human pose-based actions from sequences of depth maps. Specifically, we have designed our proposed idea into a probabilistic framework: 1) dividing each action into meaningful ordered temporal segments, 2) using pixel-neighboring intensity difference approach to identify human shape region from the confused scenes, 3) introducing a robust spatiotemporal features to extract 3D joints information via local distance features and depth human shape information to get motion variation features and 4) finally these sets of features are merged together and their feature vectors are further reduced and discriminated to get optimal vectors for real-world applications. Next, the augmented features are enhanced and symbolized by Linde-Buzo-Gray clustering algorithm for better action recognition. With these symbols, each action hidden Markov model (HMM) is trained and tested for action recognition. The experimental results demonstrated our proposed approach on three challenging depth video datasets: IM-DepthActions, MSRAction3D and MSRDailyActivity3D, producing significantly improved results over the state-of-the-art methods: especially for those actions that are not easily discernible by the existing methods.

<title>Abstract</title> The advent of wearable sensor technologies has transformed sports science and personal health monitoring, but real-time Human Activity Recognition (HAR) remains a challenge, especially in dynamic environments. Existing approaches often struggle to adapt to rapid activity transitions, handle multimodal sensor noise, and optimize for computational efficiency on resource-constrained wearable devices. In this work, we introduce an innovative system that overcomes these limitations through a novel integration of deep learning models and sensor fusion techniques. At its core, our framework features the Attention-Gated Spatiotemporal Fusion Network (AG-SFN), a deep neural network that combines Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks with a unique attention mechanism to dynamically prioritize the most relevant features from multimodal time-series data for accurate activity classification. To address sensor noise and improve data integrity, we propose the Adaptive Multimodal Kalman Fusion (AMKF) algorithm, which adapts the filtering process based on the motion intensity and sensor variance in real time. To ensure the efficient deployment of this model on wearable devices, we introduce a Dynamic Channel Pruning with Knowledge Distillation (DCP-KD) method, which significantly reduces the model size and computational load by pruning less relevant features and distilling knowledge from a large teacher model to a smaller student model. Our system was rigorously evaluated on a custom dataset of 40 participants performing six distinct sport-related activities, achieving a state-of-the-art activity recognition accuracy of 98.6%, outperforming conventional models by a significant margin. The system also demonstrated exceptional real-time performance with heart rate monitoring error reduced to 0.85 beats per minute (bpm), latency under 90 ms, and a 35% reduction in energy consumption. This research establishes a new benchmark for the development of efficient and adaptive wearable systems for real-time sports health monitoring, offering a robust solution for next-generation wearable technologies with practical applications in sports, fitness, and healthcare.