Fusing appearance and distribution information of interest points for action recognition

In this paper we propose a framework for activity recognition based on space-time interest point in video surveillance. Single type interest point feature is not sufficient to identify the activity therefore we have considered multi-class activities fussed in three dimensional (spatial & time) coordinate to achieve our objective with maximum accuracy. Our experiment shows that fusing multi class activity using global feature provides a copious representation of human daily action when compared to the use of a single feature type. This paper model the spatial temporal distribution of interest points for identification of various short or long duration activities. It is scalable in nature and work efficiently under conditions of dynamic background, changing camera view angle or zooming, front and sidelong activities. It assumes that human being is performing action. The experiment on two benchmark datasets show propitious results when compared with the state-of-the-art methods.

This paper presents a new method to detect space time interest point (STIP) from video data. We use three dimensional facet model to detect STIP and call it as facet space-time interest point or FaSTIP. The proposed algorithm detects all the desired interest points efficiently in different scales compared to other existing methods. A video clip is described as a collection of 3D wavelet base features computed at these interest points. Finally, multi-channel SVM with χ2- kernel is used to classify human actions. Our contribution here are two fold: first, we present a new algorithm for interest point detection in video data, and second, we propose a new descriptor for general human activity classification. Experimental results show the accuracy of the detected interest points and the power of descriptor compared to the state-of-the-art.

Human action recognition has been a challenging task in computer vision because of intra-class variability. State-of-the-art methods have shown good performance for constrained videos but have failed to achieve good results for complex scenes. Reasons for their failing include treating spatial and temporal dimensions without distinction as well as not capturing temporal information in video representation. To address these problems we propose principled changes to an action recognition framework that is based on video interest points (IP) detection with capturing differential motion as the central theme. First, we propose to detect points with high curl of optical flow, which captures relative motion boundaries in a frame. We track these points to form dense trajectories. Second, we discard points on the trajectories that do not represent change in motion of the same object, yielding temporally localized IPs. Third, we propose a video representation based on spatio-temporal arrangement of IPs with respect to their neighboring IPs. The proposed approach yields a compact and information-dense representation without using any local descriptor around the detected IPs. It significantly outperforms state-of-the-art methods on UCF youtube dataset, which has complex action classes, as well as on KTH dataset, which has simple action classes.

Depression is one of the most common mental health disorders with strong adverse effects on personal and social functioning. The absence of any objective diagnostic aid for depression leads to a range of subjective biases in initial diagnosis and ongoing monitoring. Psychologists use various visual cues in their assessment to quantify depression such as facial expressions, eye contact and head movements. This paper studies the contribution of (upper) body expressions and gestures for automatic depression analysis. A framework based on space-time interest points and bag of words is proposed for the analysis of upper body and facial movements. Salient interest points are selected using clustering. The major contribution of this paper lies in the creation of a bag of body expressions and a bag of facial dynamics for assessing the contribution of different body parts for depression analysis. Head movement analysis is performed by selecting rigid facial fiducial points and a new histogram of head movements is proposed. The experiments are performed on real-world clinical data where video clips of patients and healthy controls are recorded during interactive interview sessions. The results show the effectiveness of the proposed system to evaluate the contribution of various body parts in depression analysis.

Discriminative two-level feature selection for realistic human action recognition

The performance of action recognition in video sequences depends significantly on the representation of actions and the similarity measurement between the representations. In this paper, we combine two kinds of features extracted from the spatio-temporal interest points with context-aware kernels for action recognition. For the action representation, local cuboid features extracted around interest points are very popular using a Bag of Visual Words (BOVW) model. Such representations, however, ignore potentially valuable information about the global spatio-temporal distribution of interest points. We propose a new global feature to capture the detailed geometrical distribution of interest points. It is calculated by using the 3D $${\mathcal {R}}$$R transform which is defined as an extended 3D discrete Radon transform, followed by the application of a two-directional two-dimensional principal component analysis. For the similarity measurement, we model a video set as an optimized probabilistic hypergraph and propose a context-aware kernel to measure high order relationships among videos. The context-aware kernel is more robust to the noise and outliers in the data than the traditional context-free kernel which just considers the pairwise relationships between videos. The hyperedges of the hypergraph are constructed based on a learnt Mahalanobis distance metric. Any disturbing information from other classes is excluded from each hyperedge. Finally, a multiple kernel learning algorithm is designed by integrating the $$l_{2}$$l2 norm regularization into a linear SVM classifier to fuse the $${\mathcal {R}}$$R feature and the BOVW representation for action recognition. Experimental results on several datasets demonstrate the effectiveness of the proposed approach for action recognition.

Gait has been known as an effective biometric feature to identify a person at a distance, e.g., in video surveillance applications. Many methods have been proposed for gait recognitions from various different perspectives. It is found that these methods rely on appearance (e.g., shape contour, silhouette)-based analyses, which require preprocessing of foreground–background segmentation (FG/BG). This process not only causes additional time complexity, but also adversely influences performances of gait analyses due to imperfections of existing FG/BG methods. Besides, appearance-based gait recognitions are sensitive to several variations and partial occlusions, e.g., caused by carrying a bag and varying a cloth type. To avoid these limitations, this paper proposes a new framework to construct a new gait feature directly from a raw video. The proposed gait feature extraction process is performed in the spatio-temporal domain. The space-time interest points (STIPs) are detected by considering large variations along both spatial and temporal directions in local spatio-temporal volumes of a raw gait video sequence. Thus, STIPs are allocated, where there are significant movements of human body in both space and time. A histogram of oriented gradients and a histogram of optical flow are computed on a 3D video patch in a neighborhood of each detected STIP, as a STIP descriptor. Then, the bag-of-words model is applied on each set of STIP descriptors to construct a gait feature for representing and recognizing an individual gait. When compared with other existing methods in the literature, it has been shown that the performance of the proposed method is promising for the case of normal walking, and is outstanding for the case of partial occlusion caused by walking with carrying a bag and walking with varying a cloth type.

This paper presents a novel space-time feature-based human activity analysis system. We detect Space Time Interest Points (STIP) and generate their description based on the facet model. The proposed approach detects interest points in video data using the three-dimensional facet model efficiently. Then we describe each interest point by three-dimensional Haar wavelet transform and time derivatives of different order obtained from said facet model. Here we represent each video clip following the bag-of-words approach by learning feature specific dictionary. Finally, classification is done using non-linear SVM with χ 2 -kernel. We evaluate the performance of our system on standard datasets like Weizmann, KTH, UCF sports, ICD, UCF YouTube, and UCF50 and get better, or at least comparable results compared to other state-of-the-art systems.

The notion of local features in space-time has recently been proposed to capture and describe local events in video. When computing space-time descriptors, however, the result may strongly depend on the relative motion between the object and the camera. To compensate for this variation, we present a method that automatically adapts the features to the local velocity of the image pattern and, hence, results in a video representation that is stable with respect to different amounts of camera motion. Experimentally we show that the use of velocity adaptation substantially increases the repeatability of interest points as well as the stability of their associated descriptors. Moreover, for an application to human action recognition we demonstrate how velocity-adapted features enable recognition of human actions in situations with unknown camera motion and complex, non-stationary backgrounds.

Viewing angle changeable display can change the display viewing angle as needed: In the public place the display could have a narrow viewing angle for privacy, while in the private place the displays could have a wide viewing angle for the convenience of the operation and better viewing experience. This article propose a novel adjustable optical transmission device to realize the viewing angle changes for LCD by using the principle of guest- host effect of liquid crystal. The major technology is to insert a special equipment between the backlight and the LCD, through which the backlight will display either parallel or scattered features to get an either narrow or wide viewing angle. The equipment is an adjustable transmission cell (ATC) which is actually a black G-H LC cell. This ATC is the main focus of our invention. The ATC consists of a polarizer sheet and a special guest-host liquid crystal device filled with the two-phase dye (called as GH-LC in this report), to achieve the viewing angle change in the LCD. When an electrical field charges to the ATC, only the so-called near-axis lights can pass through the ATC within a relatively small angle, while the other scattered lights are absorbed sequentially by GH-LC and the polarizer sheet. On the other hand, when there is no electrical charge to the ATC, the cell behaves like a normal polarizer; and the scattered light can pass through the cell and polarizer in a normal way. This paper describes the principle and structure of the device, applies the electric field on the sample to observe the electro-optical properties, combine the theoretical and experimental research, getting the viewing angle effects of the display.

Real-world videos often contain dynamic backgrounds and evolving people activities, especially for those web videos generated by users in unconstrained scenarios. This paper proposes a new visual representation, namely scene aligned pooling, for the task of event recognition in complex videos. Based on the observation that a video clip is often composed with shots of different scenes, the key idea of scene aligned pooling is to decompose any video features into concurrent scene components, and to construct classification models adaptive to different scenes. The experiments on two large scale real-world datasets including the TRECVID Multimedia Event Detection 2011 and the Human Motion Recognition Databases (HMDB) show that our new visual representation can consistently improve various kinds of visual features such as different low-level color and texture features, or middle-level histogram of local descriptors such as SIFT, or space-time interest points, and high level semantic model features, by a significant margin. For example, we improve the-state-of-the-art accuracy on HMDB dataset by 20% in terms of accuracy.KeywordsLocal Binary PatternVector QuantizationSift FeatureVideo EventWedding CeremonyThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Due to occlusion, lighting condition, variation in clothing dance video classification is a challenging problem in computer vision domain. In this paper we present a local spatiotemporal feature model on manifold for Indian Classical Dance (ICD) classification. We represent features at each space-time interest point as a covariance matrix by fusing different order spatial and temporal derivatives. Each video clip is then represented in bag-of-words framework on manifold using Jensen-Bregman LogDet Divergence. Classification is done by popular non-linear SVM with ?2-kernel. We evaluate our system on a ICD dataset created from YouTube and get 69.39% accuracy which is better than that of the state-of-the-art human activity classification algorithms. We have also tested our algorithms on human activity benchmark datasets like KTH, and UCF50 and get promising results compared to the state-of-the-art methods.

Automatic action recognition in videos is a challenging computer vision task that has become an active research area in recent years. Existing strategies usually use kernel-based learning algorithms that considers a simple combination of different features completely disregarding how such features should be integrated to fit the given problem. Since a given feature is most suitable to describe a given image/video property, the adaptive weighting of such features can improve the performance of the learning algorithm. In this paper, we investigated the use of the Multiple Kernel Learning (MKL) algorithm to adaptive search for the best linear relation among the considered features. MKL is an extension of the support vector machines (SVMs) to work with a weighted linear combination of several single kernels. This approach allows to simultaneously estimate the weights for the multiple kernels combination as well as the underlying SVM parameters. In order to prove the validity of the MKL approach, we considered a descriptor composed of multiple features aligned with dense trajectories. We experimented our approach on a database containing 36 cooking actions. Results confirm that the use of MKL improves the classification performance.

Visual-based human action recognition is currently one of the most active research topics in computer vision. The feature representation directly has a crucial impact on the performance of the recognition. Feature representation based on bag-of-words is popular in current research, but the spatial and temporal relationship among these features is usually discarded. In order to solve this issue, a novel feature representation based on normalized interest points is proposed and utilized to recognize the human actions. The novel representation is called super-interest point. The novelty of the proposed feature is that the spatial-temporal correlation between the interest points and human body can be directly added to the representation without considering scale and location variance of the points by introducing normalized points clustering. The novelty concerns three tasks. First, to solve the diversity of human location and scale, interest points are normalized based on the normalization of the human region. Second, to obtain the spatial-temporal correlation among the interest points, the normalized points with similar spatial and temporal distance are constructed to a super-interest point by using three-dimensional clustering algorithm. Finally, by describing the appearance characteristic of the super-interest points and location relationship among the super-interest points, a new feature representation is gained. The proposed representation formation sets up the relationship among local features and human figure. Experiments on Weizmann, KTH, and UCF sports dataset demonstrate that the proposed feature is effective for human action recognition.

The problem of activity recognition is a topic that has been explored in the field of ubiquitous computing, the popularization of sensors on the most diverse types has been instrumental in improving the effectiveness of recognition. Smartphones offer a range of sensors (GPS, Accelerometer, Gyroscopes, etc.) that can be used to provide data for this type of problem. This work proposes a new model of activity recognition in GPS captured data and enriched with POIs (Points Of Interest) and user profile. The experiment was performed by 10 volunteers collecting data for 10 days. The model aims to recognize 13 different activities, divided into stop activities (Bank, breakfast, dining, lunch, praying, recreation, shopping, studying, waiting transport and working) and moves activities (in a car, on a bus and walking). The model was tested and compared to J48, SVM, ANN and RF algorithms, and obtained 97.4% hits.