Random-sampling-based spatial-temporal feature for consumer video concept classification

We propose an end-to-end system for text detection and recognition in natural scenes and consumer videos. Maximally Stable Extremal Regions which are robust to illumination and viewpoint variations are selected as text candidates. Rich shape descriptors such as Histogram of Oriented Gradients, Gabor filter, corners and geometrical features are used to represent the candidates and classified using a support vector machine. Positively labeled candidates serve as anchor regions for word formation. We then group candidate regions based on geometric and color properties to form word boundaries. To speed up the system for practical applications, we use Partial Least Squares approach for dimensionality reduction. The detected words are binarized, filtered and passed to a hidden Markov model based Optical Character Recognition (OCR) system for recognition. We show significant improvement in text detection and recognition tasks over previous approaches on a large consumer video dataset. Furthermore, the event detection system built upon the OCR output of this approach outperformed multiple other OCR-only based submissions in the recently concluded NIST TRECVID 2013 multimedia event detection evaluations.

The purpose of video object segmentation is to automatically extract objects of interest from consumer videos. This paper investigates this problem from a novel perspective of human visual attention. We roughly classify visual attentions in video scenes into two categories: static attention and dynamic attention. The static attention model is mainly responsible for segmenting the interesting objects without motion, whereas the dynamic attention model plays an important role in obtaining the interesting objects with motion. The fusion of both models allows us to obtain all interesting objects using a unified framework. This framework is easy to implement and has the great promise to become a basic tool for many content-based consumer video applications. Experimental results demonstrate the good performance of our algorithm.

For the past 2 decades, detection of buried explosive hazard has been studied extensively and several machine learning algorithms have been developed and adapted to this application. First, a pre-screener is used to identify areas of interest or alarms. Each alarm consists of a 3D data cube that corresponds to spatial down-track, crosstrack, and time respectively. Ground truth information is then used to label each alarm and generate labeled data to train a classifier to discriminate between targets and clutter objects. One of the main challenges in this approach is localizing the true depth of the alarm. On one hand, the buried object signature is not expected to cover all the depth values and extracting one global feature from all depth bins may not discriminate between object and clutter signatures effectively. On the other hand, depth ground truth is not available as this depends on the target type and size, soil properties, and other environmental conditions. Moreover, visually inspecting each alarm to select the optimal depth location is tedious, ambiguous, and not practical for very large training data. Two different approaches have been considered to train learning algorithms. The first one uses simple rules and machine learning algorithms to automate the selection of the optimal depth(s) for each alarm. The second approach avoids the labeling at the depth level and uses multiple instance learning (MIL) algorithms. In this context, each alarm is represented by a bag of multiple instances. Each instance corresponds to a feature extracted at a different depth. Since labels are needed at the bag level and not at the instance level, MIL does not require true depth information. In this paper, we propose a large-scale evaluation of the two approaches. For the first approach, we consider three methods to identify optimal depth locations and analyze their effect on the KNN and SVM classifiers. For the second approach, we consider four MIL algorithms that do not require depth information. For our analysis, we use large data collections accumulated across multiple dates and multiple test sites by a vehicle mounted downward looking ground penetrating radar (GPR) sensor. The data include a diverse set of buried explosive objects of varying shapes, metal content, and underground burial depths. Performance of all algorithms is analyzed using receiver operating characteristics (ROC).

Video data provides a rich source of information that is available to us today in large quantities e.g. from on-line resources. Tasks like segmentation benefit greatly from the analysis of spatio-temporal motion patterns in videos and recent advances in video segmentation has shown great progress in exploiting these addition cues. However, observing a single video is often not enough to predict meaningful segmentations and inference across videos becomes necessary in order to predict segmentations that are consistent with objects classes. Therefore the task of video co-segmentation is being proposed, that aims at inferring segmentation from multiple videos. But current approaches are limited to only considering binary foreground/background segmentation and multiple videos of the same object. This is a clear mismatch to the challenges that we are facing with videos from online resources or consumer videos. We propose to study multi-class video co-segmentation where the number of object classes is unknown as well as the number of instances in each frame and video. We achieve this by formulating a non-parametric Bayesian model across videos sequences that is based on a new videos segmentation prior as well as a global appearance model that links segments of the same class. We present the first multi-class video co-segmentation evaluation. We show that our method is applicable to real video data from online resources and outperforms state-of-the-art video segmentation and image co-segmentation baselines.

Weakly supervised video anomaly detection is generally formulated as a multiple instance learning (MIL) problem, where an anomaly detector learns to generate frame-level anomaly scores under the supervision of MIL-based video-level classification. However, most previous works suffer from two drawbacks: 1) they lack ability to model temporal relationships between video segments and 2) they cannot extract sufficient discriminative features to separate normal and anomalous snippets. In this article, we develop a weakly supervised temporal discriminative (WSTD) paradigm, that aims to leverage both temporal relation and feature discrimination to mitigate the above drawbacks. To this end, we propose a transformer-styled temporal feature aggregator (TTFA) and a self-guided discriminative feature encoder (SDFE). Specifically, TTFA captures multiple types of temporal relationships between video snippets from different feature subspaces, while SDFE enhances the discriminative powers of features by clustering normal snippets and maximizing the separability between anomalous snippets and normal centers in embedding space. Experimental results on three public benchmarks indicate that WSTD outperforms state-of-the-art unsupervised and weakly supervised methods, which verifies the superiority of the proposed method.

Content-based test video corpora usually builds on top of professional material or controlled settings. However, recent years have shown strong increase in user-generated content on the web. The increase in content volume creates challenges in accessibility and utility of the video content. In order to improve the utility of the user-generated videos, better automation for content-based descriptions are needed. Proper test sets are required to develop robust methods for content analysis. Detecting people from video is a common feature that is often seen in both in science and in commercial services. Unfortunately there is a lack of test data for person tracking from consumer videos. In this paper, we introduce a novel video dataset to accommodate this shortage. The dataset is done with two consumer-priced devices: a handheld camcorder and a mobile phone. Both devices were used to store material in indoor and outdoor settings with different attention levels from the people being filmed. The dataset comes with ground truth data that includes person head trajectories and other people marked in the background in MPEG-7-based metadata model. We give description of this metadata model and publish an annotation tool we used for creating the ground truth data. Also, we provide experimental results as a benchmark for all those who would like to use our dataset.

In this chapter, we show an autonomous learning scheme to automatically build visual semantic concept models from video sequences or the searched data of Internet search engines without any manual labeling work. First of all, system users specify some specific concept models to be learned automatically. Example videos or images can be obtained from the large video databases based on the result of keyword search on the automatic speech recognition transcripts. Another alternative method is to gather them by using the Internet search engines. Then, we propose to model the searched results as a term of “Quasi-Positive Bags” in the Multiple-Instance Learning (MIL). We call this as the generalized MIL (GMIL). In some of the scenarios, there is also no “Negative Bags” in the GMIL. We propose an algorithm called “Bag K-Means” to find out the maximum Diverse Density (DD) without the existence of negative bags. A cost function is found as K-Means with special “Bag Distance”. We also show a solution called “Uncertain Labeling Density” (ULD) which describes the target density distribution of instances in the case of quasi-positive bags. A “Bag Fuzzy K-Means” is presented to get the maximum of ULD. Utilizing this generalized MIL with ULD framework, the model for a particular concept can then be learned through general supervised learning methods. Experiments show that our algorithm get correct models for the concepts we are interested in.

In this paper, a model of structure semantics of image is proposed. We describe the detailed process of mathematical inference for the whole system. The model applies the knowledge of topology, neural networks, and psychology. The system first builds a structure space T using the structure parameters extracted from an image or video segment, then T is mapped to a structure information space IT by a transformation function set phi which is based on psychometric functions. The model introduces a new regulation based on psychology and neural networks for semantic analysis of image or video databases, and we call it psychological response neural network

Rapid increase in video databases has forced the industry to have efficient and effective frameworks for video retrieval and indexing. Video segmentation into scenes is widely used for video summarization, partitioning, indexing and retrieval. In this paper, we propose a framework for scene detection mainly based on entropy and Speeded Up Robust Features (SURF) features. First, we detect the fade and abrupt boundaries based on frame entropy analysis and SURF features matching. Fade boundaries are smart indication of scenes beginning or ending in many videos and dramas, and are detected by frame entropy analysis. Before abrupt boundary detection, unnecessary frames which are obviously not abrupt boundaries, such as blank screens, high intensity influenced images, sliding credits, are removed. Candidate boundaries are detected to make SURF features efficient for abrupt boundary detection, and SURF features between candidate boundaries and their adjacent frames are used to detect the abrupt boundaries. Second, key frames are extracted from abrupt shots. We evaluate our key frame extraction with other famous algorithms and show the effectiveness of the key frames. Finally, scene boundaries are detected using sliding window of size K over the key frames in temporal order. In experimental evaluation on the TRECVID-2007 shot boundary test set, the algorithm for shot boundary achieves substantial improvements over state-of-the-art methods with the precision of 99% and the recall of 97.8%. Experimental results for video segmentation into scenes are also promising, compared to famous state-of-the-art techniques.

Video caption detection and extraction is an important step for information retrieval in video databases. In this paper, we extract text information in video by fully utilizing the temporal information contained in the video. First we create a binary abstract sequence from a video segment. By analyzing the statistical pixel changes in the sequence, we can effectively locate the (dis)appealing frames of captions. Finally we extract the captions to create a summary of the video segment.

Background subtraction is the algorithmic process that segments out the region of interest often known as foreground from the background. Extensive literature and numerous algorithms exist in this domain, but most research have focused on videos captured by static cameras. The proliferation of portable platforms equipped with cameras has resulted in a large amount of video data being generated from moving cameras. This motivates the need for foundational algorithms for foreground/background segmentation in videos from moving cameras. In this dissertation, I propose three new types of background subtraction algorithms for moving cameras based on appearance, motion, and a combination of them. Comprehensive evaluation of the proposed approaches on publicly available test sequences show superiority of our system over state-of-the-art algorithms. The first method is an appearance-based global modeling of foreground and background. Features are extracted by sliding a fixed size window over the entire image without any spatial constraint to accommodate arbitrary camera movements. Supervised learning method is then used to build foreground and background models. This method is suitable for limited scene scenarios such as Pan-Tilt-Zoom surveillance cameras. The second method relies on motion. It comprises of an innovative background motion approximation mechanism followed by spatial regulation through a Mega-Pixel denoising process. This work does not need to maintain any costly appearance models and is therefore appropriate for resource constraint ego-vision systems. The proposed segmentation combined with skin cues is validated by a novel application on authenticating hand-gestured signature captured by wearable cameras. The third method combines both motion and appearance. Foreground probabilities are jointly estimated by motion and appearance. After the mega-pixel denoising process, the probability estimates and gradient image are combined by Graph-Cut to produce the segmentation mask. This method is universal as it can handle all types of moving cameras.

Despite the availability of a huge amount of video data accompanied by descriptive texts, it is not always easy to exploit the information contained in natural language in order to automatically recognize video concepts. Towards this goal, in this paper we use textual cues as means of supervision, introducing two weakly supervised techniques that extend the Multiple Instance Learning (MIL) framework: the Fuzzy Sets Multiple Instance Learning (FSMIL) and the Probabilistic Labels Multiple Instance Learning (PLMIL). The former encodes the spatio-temporal imprecision of the linguistic descriptions with Fuzzy Sets, while the latter models different interpretations of each description's semantics with Probabilistic Labels, both formulated through a convex optimization algorithm. In addition, we provide a novel technique to extract weak labels in the presence of complex semantics, that consists of semantic similarity computations. We evaluate our methods on two distinct problems, namely face and action recognition, in the challenging and realistic setting of movies accompanied by their screenplays, contained in the COGNIMUSE database. We show that, on both tasks, our method considerably outperforms a state-of-the-art weakly supervised approach, as well as other baselines.

Concept classification is important to access visual information on the level of objects and scene types. So far, intensity-based features have been widely used. To increase discriminative power, color features have been proposed only recently. As many features exist, a structured overview is required of color features in the context of concept classification.Therefore, this paper studies 1. the invariance properties and 2. the distinctiveness of color features in a structured way. The invariance properties of color features with respect to photometric changes are summarized. The distinctiveness of color features is assessed experimentally using an image and a video benchmark: the PASCAL VOC Challenge 2007 and the Mediamill Challenge.Because color features cannot be studied independently from the points at which they are extracted, different point sampling strategies based on Harris-Laplace salient points, dense sampling and the spatial pyramid are also studied.From the experimental results, it can be derived that invariance to light intensity changes and light color changes affects concept classification. The results reveal further that the usefulness of invariance is concept-specific.

Objectives: To identify a cervical cytology cell image as a representative of the normal or malignant cancerous cell. This work may be used to decide the stage of cervical cancer. Methods and Analysis: Images from the Hervlev university database have been used. Features of nucleus and cytoplasm of cervical cell images are extracted after preprocessing and segmentation. Size, shape and texture features are extracted. Appropriate and adequate features are selected by mining techniques. This feature set is fed to classifiers to decide nature of cells. Based on parameters like true positive rate, true negative rate and precision, a comparative analysis of classifiers is done. The best classifier that clearly discriminates the cells is determined for cervical cancer diagnosis. Findings: Dimensionality reduction of feature set can be done with data mining technique named as Correlation based feature subset selection. Reduced feature set can be used for classification by classifiers like Multilayer perceptron, Bayes classifiers and SVM classifier. Kappa Statistics for binary classifiers is 1 for Multilayer perceptron, 0.9968 for Binary SMO and Bayesnet, 0.9915 for NaiveBayes Classifiers. The same for seven group classifiers is 1 for LibSVM, 0.8535 for Multilayer perceptron, 0.8464 for attribute selected classifier and 0.7286 for Bagging classifier. True positive rate is 1 for Multilayer perceptron, 0.996 for Binary SMO and BayesNet, 0.975 for NaiveBayes classifier. Multilayer perceptron can be used as a classifier in both binary and seven class classifications. Improvement: Existing methods use more features for classification. Here even in the case of seven class classifier, only 11 features are used and desired result has been achieved.

This paper presents an analysis process of lecture video retrieval with automated video indexing and video search for lecture video databases. The video retrieval extracts the relevant metadata from the two main parts of lecture videos, namely the visual screen and audio tracks. From the visual screen, we firstly detect the slide transition and extract each unique slide frame with its temporal transition considered as the video segment. The textual metadata from slide frames is extracted and then recognized using video OCR technique. Based on OCR results, the corresponding text in video information is saved. Secondly, the speech-to-text analysis is carried out from the audio signal. Sphinx speech recognition models are used for recognition of speech-to-text conversion process. In the proposed work, the combined text information from the visual slide frame and audio signal were used to retrieve the video from the lecture video database. The performance of this combined video retrieval system shows a significant improvement in performance when compared with an individual system built using audio and text in video systems respectively.