Human Action Recognition using Bio-inspired Visual Motion Estimation

Activity and action detection, tracking and recognition are very demanding research area in computer vision and human computer interaction. In this paper, a video-based novel approach for human activity recognition is presented using robust hybrid features and embedded Hidden Markov Models. In the proposed HAR framework, depth maps are analyzed by temporal motion identification method to segment human silhouettes from noisy background and compute depth silhouette area for each activity to track human movements in a scene. Several representative features, including invariant, depth sequential silhouettes and spatiotemporal body joints features were fused together to explore gradient orientation change, intensity differentiation, temporal variation and local motion of specific body parts. Then, these features are processed by the dynamics of their respective class and learned, trained and recognized with specific embedded HMM having active feature values. Our experiments on two depth datasets demonstrate that the proposed features are efficient and robust over the state of the arts features for human activity recognition especially when there are similar postures of different activities.

Image processing techniques have been used in processing of potential fields data in order to delineate the edges of the anomalous sources. In this research, firstly the 2D Gabor filter was introduced. Then its operation was investigated on the three magnetized prisms which were at the three different depths and orientations. The effects of body wavelength, standard deviation and body orientation in the Gabor filter on the detection of edge bodies were studied. Investigations showed that the Gabor filter is a low pass filter. Afterwards, the Gabor filter was applied on the aeromagnetic data of Khoram Abad region in the southwest and ground magnetic data from Kashmar region in the northeast of Iran. The results show that the trend of geological structures is NW-SE which is in concordance to those obtained from other filters as well as geological map of the region. 1 2D Gabor filter The boundary of the geological structures can be determined by processing of potential field spectrums .Most filters which have been used for edge detection are based on the different degree of derivatives. The derivatives cause noises and signals amplify together which decrease resolution of edges. Therefore, using filters without any derivative terms is a prime objective. The Gabor filter has not any derivative terms in its mathematical structure. Denis Gabor first introduced the filter for image processing. This filter is linear and has been used for analyzing details of an image or its texture. The characteristics of the Gabor filter is to obtain special frequencies in a certain point or a region. Mostly this filter has been used for analyzing human vision system. The Gabor filter has two parts, real and imaginary in exponential forms which are multiplied with a Gaussian function. The real part has been used for data filtering. The Gabor filter response in spatial domain is defined by convolution of data in the Gabor matrix. The Gabor filter is a low pass filter which attenuates noises while it detects the edges of the deep anomalies in a certain direction. The Gabor filter in mathematics is in the group of filter transformation. On this basis, this filter can process images in different directions and frequencies. The Gabor filter has many applications in image processing such as texture analysis, fingerprint detection, edge detection, document processing and so on. The 2D Gabor filter was also used for identification of the edges of the geological structures and faults. For determination of the geological trends and faults, we must apply the Gabor filter in different directions and frequencies and standard deviations. In this article, we first introduce the Gabor filter and its characteristics. Afterwards this filter is applied on the aeromagnetic data from Khoram Abad and ground magnetic from an iron mine at Kashmar regions in the the southwest and northeast of Iran respectively. 2 The real data We prepared the following maps from the real data in Khoram Abad region: the map of the total magnetic intensity, the reduction to the pole of aeromagnetic map and the map resulted from the application of the Gabor filter on the residual aeromagnetic map of the region. From the latter map, the trend and edges of the geological structures were identified. The other trends may be detected by changing the various parameters of the filter specially its direction. The trend of the geological structures which is revealed by the Gabor filter is northwest- southeast. If the Gabor filter is applied with the other edge filters such as derivative of tilt angle, analytic signal filters, we can identify the boundaries better. The Gabor filter on the data from Kashmar region, delineates the boundary of the iron ore body at the studied area.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

Human action recognition has been an active research area in recent years. However, building a robust human action recognition system still remains a challenging task due to the large variations in action classes, varying human appearances, illumination changes, camera motion, occlusions and background clutter. Most previous work focus on the goal of improving recognition rates. This paper describes a computationally fast votingbased approach for human action recognition, in which the action in the video sequence is recognized based on the support of the local spatio-temporal features. The proposed technique requires no parameter tuning and can produce recognition rates that are comparable to those in recent published literature. Moreover, the technique can localize the single human action in the video sequence without much additional computation. Recognition results on the KTH and Weizmann action dataset are presented.

Human action recognition (HAR)in videos is a challenging task in computer vision. Conventional methods are prone to explore the spatiotemporal or optical representations for video actions. However, optical representation might be inefficient in some real-life situations, such as object occlusion and dim light. To address this issue, this paper presents a novel approach for human action recognition by jointly exploiting video and Wi-Fi clues. We leverage the fact that Wi-Fi signals carry discriminative information of human actions, which is robust to optical limitations. To validate this innovative thought, we conceive a practical framework for HAR and setup a dataset containing both video clips and Wi-Fi Channel State Information of human actions. The 3D convolutional neural network was used to extract the video features and the statistical algorithms were used to extract radio features. A classical linear support vector machine is employed as the classifier after the video and radio feature fusion. Comprehensive experiments on this dataset achieved desirable results with the maximum improvement in accuracy by 10%. This demonstrates our promising findings: with the aid of Wi-Fi Channel State Information, the performance of the video action recognition methods can be improved significantly, even under the optical limitation.

In this paper, a stereo camera-based novel approach for Human Activity Recognition (HAR) is presented using robust 3-D human body joint features and joint-specific Hidden Markov Models (HMMs). At f...

This paper describes an action recognition method based on the 3D local binary dense micro-block difference. The proposed algorithm is a three-stage procedure: (a) image preprocessing using a 3D Gabor filter, (b) a descriptor calculation using 3D local binary dense micro-block difference with skeleton points, and (c) SVM classification. The proposed algorithm is based on capturing 3D sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes. For intensifies motion used the convolution with a bank of 3D arbitrarily-oriented Gabor filters. We calculate the local features for pre-processed frames, such as 3D local binary dense micro-block difference (3D LBDMD). We evaluate the proposed approach on the UCF101 database. Experimental results demonstrate the effectiveness of the proposed approach on video with a stochastic textures background with comparisons of the state-of-the-art methods.

This paper presents a new approach for human action recognition around sleeping with the human body parts locations and the positional relationship between human and sleeping environment. Body parts are estimated from the depth image obtained by a time-of-flight (TOF) sensor using oriented 3D normal vector. Issues in action recognition of sleeping situation are the demand of availability in darkness, and hiding of the human body by duvets. Therefore, the extraction of image features is difficult since color and edge features are obscured by covers. Thus, first in our method, positions of four parts of the body (head, torso, thigh, and lower leg) are estimated by using the shape model of bodily surface constructed by oriented 3D normal vector. This shape model can represent the surface shape of rough body, and is effective in robust posture estimation of the body hidden with duvets. Then, action descriptor is extracted from the position of each body part. The descriptor includes temporal variation of each part of the body and spatial vector of position of the parts and the bed. Furthermore, this paper proposes hierarchical action classes and classifiers to improve the indistinct action classification. Classifiers are composed of two layers, and recognize human action by using the action descriptor. First layer focuses on spatial descriptor and classifies action roughly. Second layer focuses on temporal descriptor and classifies action finely. This approach achieves a robust recognition of obscured human by using the posture information and the hierarchical action recognition.

ABSTRACTIn the present work, we have proposed an efficient approach for human action recognition (HAR) from silhouette image sequence in videos. The efficiency of the approach lies in feature extraction and action classification. The proposed approach includes scale-shift normalization and distorted silhouette removal for the extraction of newly introduced spatiotemporal features coined as active region energy feature (AREF), and trajectory analysis. On the other hand, classification is done using hierarchical structure. An active region is the changing region in two consecutive silhouettes to accomplish the action. The AREF is estimated using active region energy image (AREI), which embraces the energies of active regions. The higher values in the AREI signify the more activeness (changing) of that region across the silhouette sequences; i.e. the region is used more (active) to complete the action. The silhouette normalization technique makes the feature extraction more robust and scale invariant. Also, the proposed approach uses a low-dimensional feature vector, which makes the whole procedure effective regarding cost in terms of timing requirement. The experimental results on publicly available Weizmann and MuHVAi data-sets clearly validate the efficiency of the proposed technique on that of the related research work regarding accuracy in the human action detection.

In this paper, we present a “key segment” mining approach for human action recognition. Our model is able to locate discriminative segments for action samples via multiple instance learning. Moreover, we propose a dynamic pooling approach to automatically find the optimal length of segment for each action sample. In addition, an effective feature is proposed for action recognition with 3D skeleton joints. It can effectively capture informative motion and shape cues of skeletons, and leads to a compact and discriminative representation. The experimental results validate the effectiveness of the proposed human action recognition method on two benchmark datasets (i.e., MSR Action3D and UCF-Kinect). Moreover, our method demonstrates superior accuracy than previous methods of using only skeleton data on MSR Action3D, and achieves the state-of-the-art performance on UCF-Kinect.Keywordsaction recognitionmultiple instance learningskeleton featuredynamic pooling

1. Using the two-dimensional (2D) spatial and spectral response profiles described in the previous two reports, we test Daugman's generalization of Marcelja's hypothesis that simple receptive fields belong to a class of linear spatial filters analogous to those described by Gabor and referred to here as 2D Gabor filters. 2. In the space domain, we found 2D Gabor filters that fit the 2D spatial response profile of each simple cell in the least-squared error sense (with a simplex algorithm), and we show that the residual error is devoid of spatial structure and statistically indistinguishable from random error. 3. Although a rigorous statistical approach was not possible with our spectral data, we also found a Gabor function that fit the 2D spectral response profile of each simple cell and observed that the residual errors are everywhere small and unstructured. 4. As an assay of spatial linearity in two dimensions, on which the applicability of Gabor theory is dependent, we compare the filter parameters estimated from the independent 2D spatial and spectral measurements described above. Estimates of most parameters from the two domains are highly correlated, indicating that assumptions about spatial linearity are valid. 5. Finally, we show that the functional form of the 2D Gabor filter provides a concise mathematical expression, which incorporates the important spatial characteristics of simple receptive fields demonstrated in the previous two reports. Prominent here are 1) Cartesian separable spatial response profiles, 2) spatial receptive fields with staggered subregion placement, 3) Cartesian separable spectral response profiles, 4) spectral response profiles with axes of symmetry not including the origin, and 5) the uniform distribution of spatial phase angles. 6. We conclude that the Gabor function provides a useful and reasonably accurate description of most spatial aspects of simple receptive fields. Thus it seems that an optimal strategy has evolved for sampling images simultaneously in the 2D spatial and spatial frequency domains.

Human Action Recognition (HAR)is a challenging time series classification problem that has received significant attention from computer vision researchers. In this paper, different techniques used for human activities are investigated, and a human action recognition approach using a Convolutional Neural Network (CNN) with Long Short Term Memory (LSTM) and generative adversarial network is proposed. The proposed research evaluates the performance of cross-entropy and adversarial loss function for HAR analysis. Two different datasets UFC101 and the classic KTH dataset, are used for experimental purposes. The UFC101 dataset contains 13k videos in which 101 human actions are included i.e., playing instrument, makeup, etc. In contrast, KTH dataset contains 600 videos containing six human activities, including walking, running, jogging, hand clapping and hand waving, performed by 25 different persons. Also, demonstrates the process of HAR by mixing both datasets and evaluate the performance. The GAN enhances the model robustness by applying adversarial training which fully discovers the underlying connections in both intra-view and cross-view aspects.

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

With the rapid increase in digital technology, most research areas are involved in human activity recognition, which can help to analyze the activities of patients. A novel approach for human activity recognition in egocentric video has been invoked in this research article. Generally, only the objects are identified, but the actions are not recognized. With this motivation and new trends, this paper presents an efficient technique to recognize the activities. In our approach, first the various activity dataset is trained, and the feature vector values are stored for various activities, which are applied to the testing inputs. Here, we use a filtering technique, i.e., a median filter followed by a segmentation method using watershed and feature extraction, such as a Histogram of Oriented Gradient (HOG), Color and GiST and a combination of all Features. Features are reduced using a genetic algorithm, and classification is done using Support Vector Machine (SVM) and a Random Forest classifier. The experimental results demonstrate that the Random Forest classifier outperformed the SVM classifier.

Recognizing human activities is one of the crucial capabilities that a robot needs to have to be useful around people. Although modern robots are equipped with various types of sensors, human activity recognition (HAR) still remains a challenging problem, particularly in the presence of noisy sensor data. In this work, we introduce a multimodal graphical attention-based HAR approach, called Multi-GAT, which hierarchically learns complementary multimodal features. We develop a multimodal mixture-of-experts model to disentangle and extract salient modality-specific features that enable feature interactions. Additionally, we introduce a novel message-passing based graphical attention approach to capture cross-modal relation for extracting complementary multimodal features. The experimental results on two multimodal human activity datasets suggest that Multi-GAT outperformed state-of-the-art HAR algorithms across all datasets and metrics tested. Finally, the experimental results with noisy sensor data indicate that Multi-GAT consistently outperforms all the evaluated baselines. The robust performance suggests that Multi-GAT can enable seamless human-robot collaboration in noisy human environments.