Recognition of Complex Events: Exploiting Temporal Dynamics between Underlying Concepts

Complex event recognition is the problem of recognizing events in long and unconstrained videos. Examples of such events are birthday party, changing a tire, making a fire …etc. In this extremely challenging task, concepts have recently shown a promising direction, where core low-level events referred to as concepts are annotated and modelled using a portion of the training data, then each complex event is described using concept scores, which are features representing the occurrence confidence for the concepts in the event. However, because of the complex nature of the videos, both the concept models and the corresponding concept scores are significantly noisy. In order to address this problem, we discuss a low-rank formulation, which combines the precisely annotated videos used to train the concepts, with the rich concept scores. The approach presented in this chapter finds a new representation for each event, which is not only low-rank, but also constrained to adhere to the concept annotation, thus suppressing the noise, and maintaining a consistent occurrence of the concepts in each event. Extensive experiments on large scale real world dataset TRECVID Multimedia Event Detection 2011 and 2012 demonstrate that the approach consistently improves the discriminativity of the concept scores by a significant margin.KeywordsComplex Event RecognitionConcept ScoresOccurrence ConfidenceUnconstrained VideosLarge-scale Real-world DatasetsThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Ongoing advances in experimental technique are making commonplace simultaneous recordings of the activity of tens to hundreds of cortical neurons at high temporal resolution. Latent population models, including Gaussian-process factor analysis and hidden linear dynamical system (LDS) models, have proven effective at capturing the statistical structure of such data sets. They can be estimated efficiently, yield useful visualisations of population activity, and are also integral building-blocks of decoding algorithms for brain-machine interfaces (BMI). One practical challenge, particularly to LDS models, is that when parameters are learned using realistic volumes of data the resulting models often fail to reflect the true temporal continuity of the dynamics; and indeed may describe a biologically-implausible unstable population dynamic that is, it may predict neural activity that grows without bound. We propose a method for learning LDS models based on expectation maximisation that constrains parameters to yield stable systems and at the same time promotes capture of temporal structure by appropriate regularisation. We show that when only little training data is available our method yields LDS parameter estimates which provide a substantially better statistical description of the data than alternatives, whilst guaranteeing stable dynamics. We demonstrate our methods using both synthetic data and extracellular multi-electrode recordings from motor cortex.

In the context of computer vision, most of the traditional action recognition techniques assign a single label to a video after analyzing the whole video. We believe that under- standing of the visual world is not limited to recognizing a specific action class or individual object instances, but also extends to how those objects interact in the scene, which im- plies recognizing events happening in the scene. In this paper we present an approach for identifying complex events in videos, starting from detection of objects and simple events using a state-of-the-art object detector (YOLO). We provide a logic based representation of events by using a realization of the Event calculus that allows us to define complex events in terms of logical rules. Axioms of the calculus are encoded in a logic program under Answer Set semantics in order to reason and formulate queries over the extracted events. The applicability of the framework is demonstrated over the scenario of recognizing different kinds of kick events in soccer videos.

The pathophysiology of L-dopa-induced dyskinesias (LIDs) in Parkinson's disease (PD) remains poorly understood. The presence of superimposed LIDs clearly differentiates motor performance of dyskinetic from non-dyskinetic PD subjects when they are on medication, but here, we investigated whether their respective motor performance differs while subjects are off L-dopa medication and LIDs are not apparent. We assessed the motor performance of nine dyskinetic and ten non-dyskinetic PD subjects off L-dopa, and of ten age-matched control subjects, during a visually guided tracking task. As previous studies have suggested that linear dynamical system (LDS) models are useful to assess motor performance in PD in addition to overall tracking error, we used LDS models to assess the damping ratio parameter of motor behavior while controlling for disease severity. While overall tracking error did not significantly differ across groups, dyskinetic PD subjects demonstrated a significantly decreased mean damping ratio compared with control and non-dyskinetic PD subjects. For both groups, greater disease severity significantly predicted a lower damping ratio, but even after controlling for disease severity, the damping ratio for dyskinetic subjects was significantly lower. Our results demonstrate, somewhat counter-intuitively, that motor performance of dyskinetic and non-dyskinetic PD subjects differ, even off L-dopa when no dyskinesias are seen. A decreased damping ratio is indicative of a tendency to overshoot a target during motor performance, similar to the dysmetria found in cerebellar patients. We discuss the possibility of motor abnormalities in dyskinetic PD patients off medication in relation to altered functional cerebellar changes described in PD.

This paper focuses on the physically reliable detection of fire, where the multispectral calibration of camera and multispectral-Martin-distance-based fire detection are studied. By regarding the multispectral irradiance sequence of fire as dynamic texture and studying the sequence with the Linear Dynamic System (LDS) model, we represent the sequence with the parameters of the LDS model. The multispectral Martin distance is defined to measure the degree of difference between image sequences. The fire detection method based on the multispectral martin distance realized the combination of the dynamic texture recognition technology and the multispectral-based fire detection. Experiments are conducted on two datasets to validate the effectiveness of our method. The work of this paper made the fire detection results more physically reliable.

Bayesian networks are a concise graphical formalism for describing probabilistic models. We have provided a brief tutorial of methods for learning and inference in dynamic Bayesian networks. In many of the interesting models, beyond the simple linear dynamical system or hidden Markov model, the calculations required for inference are intractable. Two different approaches for handling this intractability are Monte Carlo methods such as Gibbs sampling, and variational methods. An especially promising variational approach is based on exploiting tractable substructures in the Bayesian network.

A machine learning approach to detecting unknown signals in time-correlated noise is presented. In the proposed approach, a linear dynamical system (LDS) model is trained to represent the background noise via expectation-maximization (EM). The negative log-likelihood (NLL) of test data under the learned background noise LDS is computed via the Kalman filter recursions, and an unknown signal is detected if the NLL exceeds a threshold. The proposed detection scheme is derived as a generalized likelihood ratio test (GLRT) for an unknown deterministic signal in LDS noise. In simple additive white Gaussian noise (AWGN), the proposed scheme reduces to an energy detector. However, experimental results on a wireless software defined radio (SDR) testbed demonstrate that the proposed scheme substantially outperforms energy detection in a time-correlated noise background.

Switching LDS-based approach for process fault detection and classification

SUMMARYDistributed hypothalamic-midbrain neural circuits orchestrate complex behavioral responses during social interactions. How population-averaged neural activity measured by multi-fiber photometry (MFP) for calcium fluorescence signals correlates with social behaviors is a fundamental question. We propose a state-space analysis framework to characterize mouse MFP data based on dynamic latent variable models, which include continuous-state linear dynamical system (LDS) and discrete-state hidden semi-Markov model (HSMM). We validate these models on extensive MFP recordings during aggressive and mating behaviors in male-male and male-female interactions, respectively. Our results show that these models are capable of capturing both temporal behavioral structure and associated neural states. Overall, these analysis approaches provide an unbiased strategy to examine neural dynamics underlying social behaviors and reveals mechanistic insights into the relevant networks.

An emerging trend in video event classification is to learn an event from a bank of concept detector scores. Different from existing work, which simply relies on a bank containing all available detectors, we propose in this paper an algorithm that learns from examples what concepts in a bank are most informative per event, which we call the conceptlet. We model finding the conceptlet out of a large set of concept detectors as an importance sampling problem. Our proposed approximate algorithm finds the optimal conceptlet using a cross-entropy optimization. We study the behavior of video event classification based on conceptlets by performing four experiments on challenging internet video from the 2010 and 2012 TRECVID multimedia event detection tasks and Columbia's consumer video dataset. Starting from a concept bank of more than thousand precomputed detectors, our experiments establish there are (sets of) individual concept detectors that are more discriminative and appear to be more descriptive for a particular event than others, event classification using an automatically obtained conceptlet is more robust than using all available concepts, and conceptlets obtained with our cross-entropy algorithm are better than conceptlets from state-of-the-art feature selection algorithms. What is more, the conceptlets make sense for the events of interest, without being programmed to do so.

The notion of "events" is extremely important in characterizing the contents of video. An event is typically triggered by some kind of change of state captured in the video, such as when an object starts moving. The ability to reason with events is a critical step toward video understanding. This article describes the findings of a recent workshop series that has produced an ontology framework for representing video events-called Video Event Representation Language (VERL) -and a companion annotation framework, called Video Event Markup Language (VEML). One of the key concepts in this work is the modeling of events as composable, whereby complex events are constructed from simpler events by operations such as sequencing, iteration, and alternation. The article presents an extensible event and object ontology expressed in VERL and discusses a detailed example of applying VERL and VEML to the description of a "tailgating" event in surveillance video.

Analytical background and techniques: discrete-time signals, systems and transforms analysis of discrete-time speech signals probability and random processes linear model and dynamic system model optimization methods and estimation theory statistical pattern recognition. Fundamentals of speech science: phonetic process phonological process. Computational phonology and phonetics: computational phonology computational models for speech production computational models for auditory speechprocessing. Speech technology in selected areas: speech recognition speech enhancement speech synthesis.

In this paper a discriminant analysis (DA) technique called accelerated generalised subclass discriminant analysis (AGSDA) and its GPU implementation are presented. This method identifies a discriminant subspace of the input space in three steps: a) Gram matrix computation, b) eigenvalue decomposition of the between subclass factor matrix, and c) computation of the solution of a linear matrix system with symmetric positive semidefinite (SPSD) matrix of coefficients. Based on the fact that the computationally intensive parts of AGSDA, i.e. Gram matrix computation and identification of the SPSD linear matrix system solution, are highly parallelisable, a GPU implementation of AGSDA is proposed. Experimental results on large-scale datasets of TRECVID for event and concept detection show that our GPU-AGSDA method combined with LSVM outperforms LSVM alone in training time, memory consumption, and detection accuracy.

Virtual sensing for dynamic industrial process based on localized linear dynamical system models with time-delay optimization

We propose an original learning approach for image classification problems. Recognizing semantic events in video requires to preliminary learn the different classes of events. This first stage is crucial since it conditions the further classification results. In video content analysis, the task is especially difficult due to the high infra-class variability and to noisy measurements. We then represent each class by the centers of several sub-classes (or clusters) thanks to a robust partitional clustering algorithm which can be applied in parallel to a (non-predefined) number of classes. Our clustering technique overcome three main limitations of standard K-means methods: sensitivity to initialization, choice of the number of clusters and influence of outliers. Moreover, it can process the training data in an incremental way. Experimental results on sports videos are reported.