Human action recognition based on locality-constrained linear coding

To reduce quantization error, preserve the manifold of local features, distinguish the ambiguous features, and model the spatial configuration of features for Bag-of-Features (BoF) model-based human action recognition, a novel feature coding method called spatially regularized and locality-constrained linear coding (SLLC) is proposed. The spatial regularization and locality constraint are involved in the feature coding phase to model the spatial configuration of features and preserve their nonlinear manifold. The action recognition experimental results on benchmark datasets show that SLLC achieves better performance than the state-of-the-art feature coding methods such as soft vector quantization, sparse coding, and locality-constrained linear coding.

Support Vector Machine (SVM) is an important machine learning algorithm widely used for the development of machine fault diagnosis systems. In this work, we use an SVM back-end classifier, with statistical features in time and frequency domain as its input, for the development of a fault diagnosis system for a rotating machine. Our baseline system is evaluated for its speed dependent and speed independent performances. In this paper, we use locality constrained linear coding (LLC) to map the input feature vectors to a higher dimensional linear space, and remove some of the speed specific dimensions to improve the speed independent performance of the fault diagnosis system. We use LLC to do the feature mapping to the higher dimensional space, and select only the k nearest neighbour basis vectors to represent the input feature vector and thus reduce/minimize the effect of speed specific factors from the input feature vector, and thus improve the speed independent performance of the fault diagnosis. We compare the performance of the LLC-SVM system for the time and frequency domain statistical features. The proposed approach has improved the overall classification accuracy by 11.81% absolute for time domain features and 10.53% absolute for frequency domain features compared to the baseline speed independent system.

Encoding human activities using multimodal wearable sensory data

Wireless capsule endoscopy (WCE) has become an effective facility to detect digestive tract diseases. To further improve the accuracy and efficiency of computer-aided diagnosis system in the detection of intestine polyps, a novel algorithm is proposed for WCE polyp detection in this paper. First, by considering the rich color information of endoscopic images, a novel local color texture feature called histogram of local color difference (LCDH) is proposed for describing endoscopic images. A codebook acquisition method which is based upon positive samples is also proposed, generating more balanced visual words with the LCDH features. Furthermore, based on locality-constrained linear coding (LLC) algorithm, a normalized variance regular term is introduced as NVLLC algorithm, which considers the dispersion degree between k nearest visual words and features in the approximate coding phase. The final image representations are obtained from using the spatial matching pyramid model. Finally, the support vector machine is employed to classify the polyp images. The WCE dataset including 500 polyp and 500 normal images is adopted for evaluating the proposed method. Experimental results indicate that the classification accuracy, sensitivity and specificity have reached 96.00%, 95.80% and 96.20%, which performances better than traditional ways. A novel method for WCE polyp detection is developed using LCDH feature descriptor and NVLLC coding scheme, which achieves a promising performance and can be implemented in clinical-assisted diagnosis of intestinal diseases.

The Vector of Locally Aggregated Descriptors (VLAD) method, developed from BOW and Fisher Vector, has got great successes in image classification and retrieval. However, the traditional VLAD only assigns local descriptors to the closest visual words in the codebook, which is a hard voting process that leads to a large quantization error. In this paper, we propose an approach to fuse VLAD and locality-constrained linear coding (LLC), compared with the original method, several nearest neighbor centers are considered when assigning local descriptors. We use the reconstruction coefficients of LLC to obtain the weights of several nearest neighbor centers. Due to the excellent representation ability of the reconstruction coefficients for local descriptors, we also combine it with VLAD coding. Experiments were conducted on the 15 Scenes, UIUC Sports Event and Corel 10 datasets to demonstrate that our proposed method has outstanding performance in terms of classification accuracy. Our approach also does not generate much additional computational cost while encoding features.

Remote sensing image scene classification (RSISC) is an active task in the remote sensing community and has attracted great attention due to its wide applications. Recently, the deep convolutional neural networks (CNNs)-based methods have witnessed a remarkable breakthrough in performance of remote sensing image scene classification. However, the problem that the feature representation is not discriminative enough still exists, which is mainly caused by the characteristic of inter-class similarity and intra-class diversity. In this paper, we propose an efficient end-to-end local-global-fusion feature extraction (LGFFE) network for a more discriminative feature representation. Specifically, global and local features are extracted from channel and spatial dimensions respectively, based on a high-level feature map from deep CNNs. For the local features, a novel recurrent neural network (RNN)-based attention module is first proposed to capture the spatial layout information and context information across different regions. Gated recurrent units (GRUs) is then exploited to generate the important weight of each region by taking a sequence of features from image patches as input. A reweighed regional feature representation can be obtained by focusing on the key region. Then, the final feature representation can be acquired by fusing the local and global features. The whole process of feature extraction and feature fusion can be trained in an end-to-end manner. Finally, extensive experiments have been conducted on four public and widely used datasets and experimental results show that our method LGFFE outperforms baseline methods and achieves state-of-the-art results.

Depth sensors open up possibilities of dealing with the human action recognition problem by providing 3D human skeleton data and depth images of the scene. Analysis of human actions based on 3D skeleton data has become popular recently, due to its robustness and view-invariant representation. However, the skeleton alone is insufficient to distinguish actions which involve human-object interactions. In this paper, we propose a deep model which efficiently models human-object interactions and intra-class variations under viewpoint changes. First, a human body-part model is introduced to transfer the depth appearances of body-parts to a shared view-invariant space. Second, an end-to-end learning framework is proposed which is able to effectively combine the view-invariant body-part representation from skeletal and depth images, and learn the relations between the human body-parts and the environmental objects, the interactions between different human body-parts, and the temporal structure of human actions. We have evaluated the performance of our proposed model against 15 existing techniques on two large benchmark human action recognition datasets including NTU RGB+D and UWA3DII. The Experimental results show that our technique provides a significant improvement over state-of-the-art methods.

Depth imaging has, through recent technological advances, become ubiquitous as products become smaller, more affordable, and more precise. Depth cameras have also emerged as a promising modality for activity recognition as they allow detection of users’ body joints and postures. Increased resolutions have now enabled a novel use of depth cameras that facilitate more fine-grained activity descriptors: The remote detection of a person’s breathing by picking up the small distance changes from the user’s chest over time. We propose in this work a novel method to model chest elevation to robustly monitor a user’s respiration, whenever users are sitting or standing, and facing the camera. The method is robust to users occasionally blocking their torso region and is able to provide meaningful breathing features to allow classification in activity recognition tasks. We illustrate that with this method, with specific activities such as paced-breathing meditating, performing breathing exercises, or post-exercise recovery, our model delivers a breathing accuracy that matches that of a commercial respiration chest monitor belt. Results show that the breathing rate can be detected with our method at an accuracy of 92 to 97% from a distance of two metres, outperforming state-of-the-art depth imagining methods especially for non-sedentary persons, and allowing separation of activities in respiration-derived features space.

The bag-of-words approach with local spatio-temporal features have become a popular video representation for action recognition in videos. Together these techniques have demonstrated high recognition results for a number of action classes. Recent approaches have typically focused on capturing global statistics of features. However, existing methods ignore relations between features and thus may not be discriminative enough. Therefore, we propose a novel feature representation which captures statistics of pairwise co-occurring local spatio-temporal features. Our representation captures not only global distribution of features but also focuses on geometric and appearance (both visual and motion) relations among the features. Calculating a set of bag-of-words representations with different geometrical arrangement among the features, we keep an important association between appearance and geometric information. Using two benchmark datasets for human action recognition, we demonstrate that our representation enhances the discriminative power of features and improves action recognition performance.

Action recognition is a challenging task due to intra-class motion variation caused by diverse style and duration in performed action videos. Previous works on action recognition task are more focused on hand-crafted features, treat different sources of information independently, and simply combine them before classification. In this paper we study action recognition from depth sequences captured by RGB-D cameras using kernel descriptors. Kernel descriptors provide an elegant way for combining a variety of information sources and can be easily applied to a hierarchical structure. We show how using kernel descriptors over pixel-level attributes in video sequences gains a great success compared to state-of-the-art methods. Following the success of kernel descriptors [1] on object recognition tasks, we employ 3D kernel descriptors, which are a unified framework for capturing pixel-level attributes and turning them into discriminative low-level features on individual 3D patches. We use efficient match kernel (EMK) [2] as the next level of our hierarchical structure to abstract the mid-level features for classification. Through extensive experiments we demonstrate using pixel-level attributes in the hierarchical architecture of our 3D kernel descriptor and EMK achieves superior performance on the standard depth sequences benchmarks.

It is challenging for an intelligent system to recognize the actions recorded in an RGB video due to the large amount of information and wide variations in the RGB video. On the other side, skeleton data focuses on the region of human body but lacks the interaction information with the background, which is complementary to the RGB data. Recently, some works focus on combining the RGB and skeleton data together to boost the performance of action recognition. However, the semantic information between joints is missing in existing works, which is important for action recognition. In this paper, we propose a novel semantic-augmented local decision aggregation network for action recognition. Specifically, we regard the area of body joints as the attention region to extract a local spatio-temporal feature for each body joint. In order to take advantage of the semantic information between joints, we propose a semantic information module, which jointly encodes the spatial and temporal index of body joints to enhance the representation ability of the local features. For better learning ability, instead of aggregating the local features, we first make decisions based on each individual local feature and then aggregate the local decisions for final recognition, which reflects the idea of resemble learning. Extensive experiments demonstrate the effectiveness of our proposed module which improves the performance of action recognition on three commonly used datasets.KeywordsAction recognitionAttentionLocal featuresDecision aggregationSemantic information

Local structure preserving discriminative projections for RGB-D sensor-based scene classification

Accurate and interpretable AI models play a critical role in medical image analysis. However, despite advancements in explainable AI (XAI), existing methods struggle with inconsistent interpretability. To overcome this limitation, we introduce the Integrated Latent and Attention Mapping (ILAM) framework, which enhances both classification accuracy and explainability by fusing local and global feature representations. ILAM integrates a custom-designed Autoencoder (AE) with a Vision Transformer (ViT), where the AE learns fine-grained local features through unsupervised patchwise image reconstruction in the latent space. These local features are then fused with global representations extracted by ViT, creating a hybrid model that improves both performance and post hoc interpretability. To refine explainability, ILAM incorporates a modified attention rollout mechanism, which recursively aggregates latent feature representations and attention weights to produce precise and stable activation maps. We evaluate ILAM on three publicly available medical imaging datasets-BreakHis, Chest X-Ray, and Retinal, demonstrating its superior performance over transformer-based models such as ViT, DeiT, CvT, and SwinT. ILAM consistently generates detailed and reliable activation maps, providing clearer visualizations of critical image regions influencing model decisions. By effectively combining local and global feature fusion, ILAM establishes itself as a robust and interpretable framework for medical image classification.

EAAR: Efficient and Accurate Action Recognition model with enhanced spatio-temporal perception.

In the analysis of histopathological images, both holistic (e.g., architecture features) and local appearance features demonstrate excellent performance, while their accuracy may vary dramatically when providing different inputs. This motivates us to investigate how to fuse results from these features to enhance the accuracy. Particularly, we employ content-based image retrieval approaches to discover morphologically relevant images for image-guided diagnosis, using holistic and local features, both of which are generated from the cell detection results by a stacked sparse autoencoder. Because of the dramatically different characteristics and representations of these heterogeneous features (i.e., holistic and local), their results may not agree with each other, causing difficulties for traditional fusion methods. In this paper, we employ a graph-based query-specific fusion approach where multiple retrieval results (i.e., rank lists) are integrated and reordered based on a fused graph. The proposed method is capable of combining the strengths of local or holistic features adaptively for different inputs. We evaluate our method on a challenging clinical problem, i.e., histopathological image-guided diagnosis of intraductal breast lesions, and it achieves 91.67% classification accuracy on 120 breast tissue images from 40 patients.