Human action recognition with 3D local binary patterns and dense micro-block difference description

The emerging research work in human activity recognition is required for smart environment especially in health and home applications. This paper proposed a bio-inspired human visual system model to extract motion features for human activity recognition. The architecture of classical V1-MT model is modified into hierarchical architecture of feedforward model; (1) three-dimensional (3D) Gabor filter in primary visual cortex (V1) layer, (2) new pooling mechanism of active motion map (3) final motion estimation based on confidence mapping in MT layer. The 3D Gabor filter used to model the spatio-temporal direction and speed tuning of the V1 cells. The active mean motion map indicate the most significant feature vector in representing human actions, while the confidence map used to estimate accurate velocity characteristic of neurons in MT layer. The approach is carried out on the Weizmann and KTH action database. The results demonstrate the effectiveness of proposed method and its advantage in terms of recognition performance and computational efficiency compared to other bio-inspired approaches for human action recognition.

Image processing techniques have been used in processing of potential fields data in order to delineate the edges of the anomalous sources. In this research, firstly the 2D Gabor filter was introduced. Then its operation was investigated on the three magnetized prisms which were at the three different depths and orientations. The effects of body wavelength, standard deviation and body orientation in the Gabor filter on the detection of edge bodies were studied. Investigations showed that the Gabor filter is a low pass filter. Afterwards, the Gabor filter was applied on the aeromagnetic data of Khoram Abad region in the southwest and ground magnetic data from Kashmar region in the northeast of Iran. The results show that the trend of geological structures is NW-SE which is in concordance to those obtained from other filters as well as geological map of the region. 1 2D Gabor filter The boundary of the geological structures can be determined by processing of potential field spectrums .Most filters which have been used for edge detection are based on the different degree of derivatives. The derivatives cause noises and signals amplify together which decrease resolution of edges. Therefore, using filters without any derivative terms is a prime objective. The Gabor filter has not any derivative terms in its mathematical structure. Denis Gabor first introduced the filter for image processing. This filter is linear and has been used for analyzing details of an image or its texture. The characteristics of the Gabor filter is to obtain special frequencies in a certain point or a region. Mostly this filter has been used for analyzing human vision system. The Gabor filter has two parts, real and imaginary in exponential forms which are multiplied with a Gaussian function. The real part has been used for data filtering. The Gabor filter response in spatial domain is defined by convolution of data in the Gabor matrix. The Gabor filter is a low pass filter which attenuates noises while it detects the edges of the deep anomalies in a certain direction. The Gabor filter in mathematics is in the group of filter transformation. On this basis, this filter can process images in different directions and frequencies. The Gabor filter has many applications in image processing such as texture analysis, fingerprint detection, edge detection, document processing and so on. The 2D Gabor filter was also used for identification of the edges of the geological structures and faults. For determination of the geological trends and faults, we must apply the Gabor filter in different directions and frequencies and standard deviations. In this article, we first introduce the Gabor filter and its characteristics. Afterwards this filter is applied on the aeromagnetic data from Khoram Abad and ground magnetic from an iron mine at Kashmar regions in the the southwest and northeast of Iran respectively. 2 The real data We prepared the following maps from the real data in Khoram Abad region: the map of the total magnetic intensity, the reduction to the pole of aeromagnetic map and the map resulted from the application of the Gabor filter on the residual aeromagnetic map of the region. From the latter map, the trend and edges of the geological structures were identified. The other trends may be detected by changing the various parameters of the filter specially its direction. The trend of the geological structures which is revealed by the Gabor filter is northwest- southeast. If the Gabor filter is applied with the other edge filters such as derivative of tilt angle, analytic signal filters, we can identify the boundaries better. The Gabor filter on the data from Kashmar region, delineates the boundary of the iron ore body at the studied area.

Texture feature has been widely used in object recognition, image content analysis and many others. Among various approaches to texture feature extraction, Gabor filter has emerged as one of the most popular ones. Gabor filter-based feature extractor is in fact a Gabor filter bank defined by its parameters including frequencies, orientations and smooth parameters of Gaussian envelope. In the literature, different parameter settings have been suggested, and filter banks created by these parameter settings work well in general. From the perspective of pattern classification, however, filter banks thus designed may not be ideal. In the present study, we propose a new approach to Gabor filter bank design, by incorporating feature selection, i.e. filter selection, into the design process. The merits of incorporating filter selection in filter bank design are twofold. Firstly, filter selection produces a compact Gabor filter bank and hence reduces computational complexity of texture feature extraction. Secondly, Gabor filter bank thus designed produces low-dimensional feature representation with improved sample-to-feature ratio, and this in turn leads to improved performance of texture classification. Experiment results on benchmark datasets and a real application have demonstrated the effectiveness of the proposed method.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

The randomness and richness present in the iris texture make the 2D Gabor filter bank analysis a suitable technique to be used for iris recognition systems. To accurately characterize complex texture structures using 2D Gabor filters it is necessary to use multiple sets of parameters of this type of filters. This paper proposes a technique of optimizing multiple sets of 2D Gabor filter parameters to gradually enhance the accuracy of an iris recognition system. The proposed methodology is suitable to be applied on both near infrared and visible spectrum iris images. To illustrate the efficiency of the filter bank design technique, UBIRISv1 database was used for benchmarking.

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Motion is the most informative cue for human action recognition. Regions with high motion saliency indicate where actions occur and contain visual information that is most relevant to actions. In this paper, we propose a novel approach for human action recognition based on oriented motion salient regions (OMSRs). Firstly, we apply a bank of 3D Gabor filters and an opponent inhibition operator to detect OMSRs of videos, each of which corresponds to a specific motion direction. Then, a new low-level feature, named as oriented motion salient descriptor (OMSD), is proposed to describe the obtained OMSRs through the statistics of the texture in the regions. Next, we utilize the obtained OMSDs to explore the oriented characteristics of action classes and generate a set of class-specific oriented attributes (CSOAs) for each class. These CSOAs provide a compact and discriminative middle-level representation for human actions. Finally, an SVM classifier is utilized for human action classification and a new compatibility function is devised for measuring how well a given action matches to the CSOAs of a certain class. We test the proposed approach on four public datasets and the experimental results validate the effectiveness of our approach.KeywordsVideo SequenceAction ClassGabor FilterHuman Action RecognitionCompatibility FunctionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

<p>Previous approaches to texture analysis and segmentation use multichannel filtering by applying a set of filters in the frequency domain or a set of masks in the spatial domain. This paper presents two new texture segmentation algorithms based on multichannel filtering in conjunction with neural networks for feature extraction and segmentation. The features extracted by Gabor filters have been applied for image segmentation and analysis. Suitable choices of filter parameters and filter bank coverage in the frequency domain to optimize the filters are discussed. Here we introduce two methods to optimize Gabor filter bank. First, a Gabor filter bank with a flat response is implemented and the optimal feature dimension is extracted by competitive networks. Second, a subset of Gabor filter bank is selected to compose the best discriminative filters, so that each filter in this small set can discriminate a pair of textures in a given image. In both approaches, multilayer perceptrons are employed to segment the extracted features. The comparisons of segmentation results generated using the proposed methods and previous research using Gabor, discrete cosine transform (DCT), and Laws filters are presented. Finally, the segmentation results generated by applying the optimized filter banks to textured images are presented and discussed. </p>

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Human Activity Recognition (HAR) is pivotal in ubiquitous computing, offering benefits to human-centric services such as health monitoring, smart homes, and eldercare systems. HAR leverages smartphones, smartwatches, and other wearable devices to collect sensory data annotated with activity labels, which are then used to train machine learning or deep learning models for automatic activity recognition. Effective HAR systems must integrate information from multiple modalities to accurately assist users. This paper introduces the XAttentionHAR model, an innovative cross-modality attention-based ensemble model for HAR. Our approach utilizes a self-attention module to extract features within each modality and an inter-domain cross-attention module to capture and integrate long-term dependencies across domains. The cross-modality attention mechanism enhances the fusion of diverse modalities, enriching the semantic information. We conducted extensive experiments on the WISDM public dataset, which includes accelerometer and gyroscope data from smartwatches and smartphones. Our results demonstrate that XAttentionHAR outperforms other state-of-the-art methods in activity recognition, achieving 98.48% accuracy for smartphone-based HAR and 98.73% accuracy for smartwatch-based HAR, paving the way for improved human-centric services.

Human motion detection and activity recognition are becoming vital for the applications in smart homes. Traditional Human Activity Recognition (HAR) mechanisms use special devices to track human motions, such as cameras (vision-based) and various types of sensors (sensor-based). These mechanisms are applied in different applications, such as home security, Human–Computer Interaction (HCI), gaming, and healthcare. However, traditional HAR methods require heavy installation, and can only work under strict conditions. Recently, wireless signals have been utilized to track human motion and HAR in indoor environments. The motion of an object in the test environment causes fluctuations and changes in the Wi-Fi signal reflections at the receiver, which result in variations in received signals. These fluctuations can be used to track object (i.e., a human) motion in indoor environments. This phenomenon can be improved and leveraged in the future to improve the internet of things (IoT) and smart home devices. The main Wi-Fi sensing methods can be broadly categorized as Received Signal Strength Indicator (RSSI), Wi-Fi radar (by using Software Defined Radio (SDR)) and Channel State Information (CSI). CSI and RSSI can be considered as device-free mechanisms because they do not require cumbersome installation, whereas the Wi-Fi radar mechanism requires special devices (i.e., Universal Software Radio Peripheral (USRP)). Recent studies demonstrate that CSI outperforms RSSI in sensing accuracy due to its stability and rich information. This paper presents a comprehensive survey of recent advances in the CSI-based sensing mechanism and illustrates the drawbacks, discusses challenges, and presents some suggestions for the future of device-free sensing technology.

In this research we concentrate on adaptive texture feature extractors which are automatically extracted from an image collection. We use independent component analysis (ICA) to extract independent component filters (ICF). ICF have previously been shown to include edge filters, having properties similar to the receptive fields of simple cells of the human visual cortex. In this thesis we evaluate the utility of ICF-based collection-specific features in the context of content-based image retrieval (CBIR), with the view to demonstrate the viability of using such automatic collection-specific features. We find that global features extracted using a small number of ICF outperform those extracted by a bank of Gabor filters, even when very large number of Gabors are used. We also present comparisons against a variety of state-of-the-art features and show that ICF-based features perform better than these, without the need for any hand-tuning. ICA extracts large number of filters. In order to find a useful smaller subset we evaluate a previously-published variance-based filter selection method. We identify the shortcomings of this method. Our proposed improvements to filter-based feature selection and extraction, response scaling and locally normalised convolution, was a result of trying to address some of these shortcomings. Even with these improvements the variance-based method has an intrinsic flaw of being susceptible to selecting redundant filters. We propose a new filter selection method based on normalised cross-correlation and clustering. We show that our method selects a more useful subset compared to the previously published variance-based method. We also propose a salient-point based local feature which uses ICF and show that our proposed feature performs better than SIFT features and also the global ICF features. We further illustrate the utility of ICFSIFT by developing a grid-based adaptive local feature and we demonstrate that ICFSIFT extracts different and useful information compared to the ICF-based global and grid-based local features. After applying the techniques proposed in this thesis to four standard texture collections, we show that, while hand-tuned features can work well for their target collection, the automatic adaptive features perform better for the general case.