Feature Weighting via Optimal Thresholding for Video Analysis

Analysis and detection of complex events in videos require a semantic representation of the video content. Existing video semantic representation methods typically require users to pre-define an exhaustive concept lexicon and manually annotate the presence of the concepts in each video, which is infeasible for real-world video event detection problems. In this paper, we propose an automatic semantic concept discovery scheme by exploiting Internet images and their associated tags. Given a target event and its textual descriptions, we crawl a collection of images and their associated tags by performing text based image search using the noun and verb pairs extracted from the event textual descriptions. The system first identifies the candidate concepts for an event by measuring whether a tag is a meaningful word and visually detectable. Then a concept visual model is built for each candidate concept using a SVM classifier with probabilistic output. Finally, the concept models are applied to generate concept based video representations. We use the TRECVID Multimedia Event Detection (MED) 2013 as our video test set and crawl 400K Flickr images to automatically discover 2, 000 visual concepts. We show significant performance gains of the proposed concept discovery method over different video event detection tasks including supervised event modeling over concept space and semantic based zero-shot retrieval without training examples. Importantly, we show the proposed method of automatic concept discovery outperforms other well-known concept library construction approaches such as Classemes and ImageNet by a large margin (228%) in zero-shot event retrieval. Finally, subjective evaluation by humans also confirms clear superiority of the proposed method in discovering concepts for event representation.

The World Wide Web has been witnessing an explosion of video content. Video data are becoming one of the most valuable sources to assess insights and information. However, existing video search methods are still based on text matching (text-to-text search), and could fail for the huge volumes of videos that have little relevant metadata or no metadata at all. In this paper, we propose an accurate, efficient and scalable semantic search method for Internet videos that allows for intelligent and flexible search schemes over the video content (text-to-video search and text&video-to-video search). To achieve this ambitious goal, we propose several novel methods to improve accuracy and efficiency. The extensive experiments demonstrate that the proposed methods are able to surpass state-of-the-art accuracy and efficiency on multiple datasets. Based on the proposed methods, we implement E-Lamp Lite, the first of its kind large-scale semantic search engine for Internet videos. According to National Institute of Standards and Technology (NIST), it achieved the best accuracy in the TRECVID Multimedia Event Detection (MED) 2013, 2014 and 2015, one of the most representative task for content-based video search. To the best of our knowledge, E-Lamp Lite is the first content-based semantic search system that is capable of indexing and searching a collection of 100 million videos.

Multimedia event detection (MED) is a retrieval task with the goal of finding videos of a particular event in a large scale internet video archive, given example videos and text descriptions. Nowadays, different multimodal fusion schemes of low-level and high-level features are extensively investigated and evaluated for MED. For most of events in MED, people are usually the central subjects in videos. The face of a person can be considered as the most important factor which brings a lot of information describing the video events. However, face information has not been systematically investigated in the previous research for MED. In this paper, we investigate the possibility of using the high-level face information to assist multimedia event detection. Moreover, since the labeled data in TRECVID MED dataset are limited, we propose a semi-supervised kernel ridge regression which works well in practice to explore the useful information from unlabeled data to assist the event detection. Extensive experimental results on TRECVID MED dataset show that our proposed method outperforms the state-of-the-art methods by up to 4%.

Event Bank based multimedia representation via latent group logistic regression minimization

In this paper, the optimal threshold for fire detection is derived based on the probability density functions of the fire and background pixels. The threshold values and the commission and omission errors are computed according to several detection criteria: maximum likelihood, null-hypothesis testing, and minimum cost. This statistical approach enables the determination of optimal threshold value according to the specific requirements of the detection tasks I. INTRODUCTION Active fire detection using satellite thermal sensors usually involves thresholding the detected brightness temperature in several bands. The frequently used features for fire detection are the brightness temperature in the 4-µm wavelength band (T4) and the brightness temperature difference between the 4- µm and 11- µm bands (∆T=T4-T11) (1), (2), (3). Due to the statistical nature of the fire and background temperature distribution, no matter how the thresholds are chosen, there are bound to be commission errors (i.e. false alarms) and omission errors (false negatives) in the detection outcomes. In the commonly used fire detection algorithms, the thresholds are often arbitrarily defined, and hence may not be optimal for a given fire detection task. The optimal threshold values are expected to be site dependent. They are different in different regions and for different fire conditions (such as smoldering and flaming fires). Hence, a global set of threshold values may not work optimally for a specific site. The task of active fire hot spot detection can be modeled as a stochastic target detection procedure (4). In this model, each pixel of the image occupies a point (T4, ∆T) in the two- dimensional feature space, and each pixel belongs to either the fire class or the non-fire background class. This approach considers the probability density functions of the fire and background pixels and optimal thresholds are derived depending on the specific objectives of the detection tasks. In a previous work (4), we implemented a method for deriving the optimal detection threshold by minimizing a cost function which is a weighted sum of the omission and commission errors. Alternatively, the threshold can also be derived based on other criteria. In this paper, we derive the optimal thresholds for detecting active fires in MODIS scenes using various detection criteria: maximum likelihood, constant false alarm rate, and null-hypothesis testing. The test data set consists of several MODIS scenes over Sumatra and Borneo islands and coincidental high resolution SPOT scenes (5). The SPOT data serves as ground truth for validation of MODIS hotspots detected by the new algorithms based on stochastic models. The probability density functions (pdf's) of the fire and background pixels are derived from this dataset and these pdf's are used to derive the optimal thresholds for active fire detection.

The state-of-the-art in example-based multimedia event detection (MED) rests on heterogeneous classifiers whose scores are typically combined in a late-fusion scheme. Recent studies on this topic have failed to reach a clear consensus as to whether machine learning techniques can outperform rule-based fusion schemes with varying amount of training data. In this paper, we present two parametric approaches to late fusion: a normalization scheme for arithmetic mean fusion (logistic averaging) and a fusion scheme based on logistic regression, and compare them to widely used rule-based fusion schemes. We also describe how logistic regression can be used to calibrate the fused detection scores to predict an optimal threshold given a detection prior and costs on errors. We discuss the advantages and shortcomings of each approach when the amount of positives available for training varies from 10 positives (10Ex) to 100 positives (100Ex). Experiments were run using video data from the NIST TRECVID MED 2013 evaluation and results were reported in terms of a ranking metric: the mean average precision (mAP) and R0, a cost-based metric introduced in TRECVID MED 2013.

Multiple depth-levels features fusion enhanced network for action recognition

Automatic extraction of road information based on data-driven methods is significant for various practical applications. Remote sensing (RS) images and GPS trajectories are two available data sources that can describe roads from a complementary perspective, and fusing them together can improve road detection performance. However, existing studies on the combination of RS images and GPS trajectories do not fully utilize their enhanced information about roads and suffer from road information loss. Moreover, roads and intersections are two crucial elements of road network generation, and they are closely related to each other. Therefore, we propose a multitask and multisource adaptive fusion (MTMSAF) network, which leverages RS images and trajectory datasets to execute road extraction tasks and intersection detection tasks simultaneously. The MTMSAF network is built on three components. First, two encoder steams are designed to capture road features from RS images and trajectories. Then, an adaptive fusion module is created to fuse the individual road features at each level in a guiding fashion. Finally, two specific decoders are proposed: one is used to implement the road region detection task by considering multidirectional geometric road information, and the other is used to perform the intersection extraction task by recovering intersection information with the road region feature. Four state-of-the-art deep learning-based methods, including segmentation networks and road extraction networks, are compared with the proposed approach. The results show the superiority of our approach for both road detection and intersection extraction tasks.

This paper describes a technique for increasing the ability of a text-dependent speaker recognition system to discriminate between speaker classes; this technique is to be performed in conjunction with the nonlinear time alignment between a reference pattern and a test pattern. Unlike the standard approach, where the training of the recognition system merely consists of storing and averaging or selecting the time normalized training patterns separately for each class, the training phase of the system is extended in that a weight is determined for each individual feature component of the complete reference pattern according to the ability of the feature to distinguish between speaker classes. The weights depend on the time axis as well as on the frequency axis. The overall distance computed after nonlinear time alignment between a reference pattern and a test pattern thus becomes a function of the given set of weights of the reference class considered. For each class, the optimum weights result from the ideal criterion of minimum error rate. Instead of this criterion, the closely related but mathematically more convenient Fisher criterion is used that leads to a closed from solution for the unknown weights. Based on these weights, the selection of subsets of effective features is studied in order to further improve the class discrimination. The feature weighting and selecting techniques are tested using a data base of utterances recorded off dialed-up telephone lines. The experiments indicate that feature weighting and feature selection can reduce the error rates by a factor of two or more both for speaker identification and speaker verification.

In mammals, the lateral geniculate nucleus (LGN) and the superior colliculus (SC) are the major targets of visual inputs from the retina. The LGN projects mainly to primary visual cortex (V1) while the SC targets the thalamus and brainstem, providing two potential pathways for processing visual inputs. Indeed, cortical lesion experiments in rodents have yielded mixed results, leading to the hypothesis that performance of simple visual behaviors may involve computations performed entirely by this subcortical pathway through the SC. However, these previous experiments have been limited by both their assays of behavioral performance and their use of lesions to change cortical activity. To determine the contribution of V1 to these tasks, we trained mice to perform threshold detection tasks in which they reported changes in either the contrast or orientation of visual stimuli. We then reversibly inhibited V1 by optogenetically activating parvalbumin-expressing inhibitory neurons with channelrhodopsin-2. We found that suppressing activity in V1 substantially impaired performance in visual detection tasks. The behavioral deficit depended on the retinotopic position of the visual stimulus, confirming that the effect was due to the specific suppression of the visually driven V1 neurons. Behavioral effects were seen with only moderate changes in neuronal activity, as inactivation that raised neuronal contrast thresholds by a median of only 14% was associated with a doubling of behavioral contrast detection threshold. Thus, detection of changes in either orientation or contrast is dependent on, and highly sensitive to, the activity of neurons in V1.

This study evaluates the use of deep learning and data sampling on a class-imbalanced Big Data problem, i.e. Medicare fraud detection. Medicare offers affordable health insurance to the elderly population and serves more than 15% of the United States population. To increase transparency and help reduce fraud, the Centers for Medicare and Medicaid Services (CMS) have made several data sets publicly available for analysis. Our research group has conducted several studies using CMS data and traditional machine learning algorithms (non-deep learning), but challenges associated with severe class imbalance leave room for improvement. These previous studies serve as baselines as we employ deep neural networks with various data-sampling techniques to determine the efficacy of deep learning in addressing class imbalance. Random over-sampling (ROS), random under-sampling (RUS), and combinations of the two (ROS-RUS) are applied to study how varying levels of class imbalance impact model training and performance. Classwise performance is maximized by identifying optimal decision thresholds, and a strong linear relationship between minority class size and optimal threshold is observed. Results show that ROS significantly outperforms RUS, combining RUS and ROS both maximizes performance and efficiency with a 4× speedup in training time, and the default threshold of 0.5 is never optimal when training data is imbalanced. To the best of our knowledge, this is the first study to provide statistical results comparing ROS, RUS, and ROS-RUS deep learning methods across a range of class distributions. Additional contributions include a unique analysis of thresholding as it relates to the minority class size and state-of-the-art performance on the given fraud detection task.

On the benefit of feature selection and ensemble feature selection for fuzzy k-nearest neighbor classification

Event detection is a crucial task in information extraction. Existing research primarily focuses on machine automatic detection tasks, which often perform poorly in certain practical applications. To address this, an interactive event-detection mode of “machine recommendation-human review–machine incremental learning” was proposed. In this mode, we study a few-shot continual class-incremental learning scenario, where the challenge is to learn new-class events with limited samples while preserving memory of old class events. To tackle these challenges, we propose a class-incremental learning method for interactive event detection via Interaction, Contrast and Distillation (ICD). We design a replay strategy based on representative and confusable samples to retain the most valuable samples under limited conditions; we introduce semantic-boundary-smoothness contrastive learning for effective learning of new-class events with few samples; and we employ hierarchical distillation to mitigate catastrophic forgetting. These methods complement each other and show strong performance. Experimental results demonstrate that, in the 5-shot 5-round class incremental-learning settings on two Chinese event-detection datasets ACE and DuEE, our method achieves final recall rates of 71.48% and 90.39%, respectively, improving by 6.86% and 3.90% over the best baseline methods.

Human performance in visual detection, discrimination, identification, and search tasks typically improves with practice. Psychophysical studies suggest that perceptual learning is mediated by an enhancement in the coding of the signal, and physiological studies suggest that it might be related to the plasticity in the weighting or selection of sensory units coding task relevant information (learning through attention optimization). We propose an experimental paradigm (optimal perceptual learning paradigm) to systematically study the dynamics of perceptual learning in humans by allowing comparisons to that of an optimal Bayesian algorithm and a number of suboptimal learning models. We measured improvement in human localization (eight-alternative forced-choice with feedback) performance of a target randomly sampled from four elongated Gaussian targets with different orientations and polarities and kept as a target for a block of four trials. The results suggest that the human perceptual learning can occur within a lapse of four trials (<1 min) but that human learning is slower and incomplete with respect to the optimal algorithm (23.3% reduction in human efficiency from the 1st-to-4th learning trials). The greatest improvement in human performance, occurring from the 1st-to-2nd learning trial, was also present in the optimal observer, and, thus reflects a property inherent to the visual task and not a property particular to the human perceptual learning mechanism. One notable source of human inefficiency is that, unlike the ideal observer, human learning relies more heavily on previous decisions than on the provided feedback, resulting in no human learning on trials following a previous incorrect localization decision. Finally, the proposed theory and paradigm provide a flexible framework for future studies to evaluate the optimality of human learning of other visual cues and/or sensory modalities.

Two-stream Convolutional Networks (ConvNets) have achieved great success in video action recognition. Research also shows that early fusion of the two-stream ConvNets can further boost the performance. Existing fusion methods focus on short snippets thus fails to learn global representations for videos. We introduce a Convolutional Gated Recurrent Units (ConvGRU) fusion method to model long-term dependency inside actions. This fusion method takes advantage of both Recurrent Neural Networks (RNN) models which have strong capacity to handle long-term dependency for sequence modeling and early fusion architecture which learns the evolution of appearance feature and motion feature. We further propose an end-to-end architecture according to this fusion method and evaluate our approach using a widely used action recognition dataset named UCF101. We investigate different input lengths and fusion layers and find that fusing at the last convolutional layer with an input length of 10 entries yields best performance (93.0%) which is comparable to the state-of-the-art.