Spatial–Temporal Context-Aware Online Action Detection and Prediction

A key factor that makes action detection in videos different from general video classification is human-guided clues, especially motion signals. Since not all the pixels in a video are informative for action recognition, the irrelevant and redundant parts can lead to a lot of noise and be burdensome for both feature extraction and classifier training. This encourages the researchers to seek out the design of the attentive model that can dynamically focus computations on the key spatiotemporal volumes. In this paper, we propose a motion-centric attention model for action detection in videos which imitates the human perception of saccade and fixation procedures while detecting actions in a video. Specifically, we first present a strategy to generate motion-centric locations based on the density peak of motion signals, providing reliable candidates around which actions have high possibilities to occur. Then, we introduce an attention model that conducts the saccade and fixation procedures on these candidates to observe local spatiotemporal visual information, preserve internal comprehension, and produce the action proposals on temporal bounds. Afterward, a classifier with several variants is prepared to classify the action proposals and decide which one to fixate and generate the final predictions. We show how to efficiently train our model to produce fast and accurate action detection, by scanning only a small fraction of locations in a video. The extensive experiments on three challenging datasets show promising results with both accuracy and speed.

We address the problem of action detection in videos. Driven by the latest progress in object detection from 2D images, we build action models using rich feature hierarchies derived from shape and kinematic cues. We incorporate appearance and motion in two ways. First, starting from image region proposals we select those that are motion salient and thus are more likely to contain the action. This leads to a significant reduction in the number of regions being processed and allows for faster computations. Second, we extract spatio-temporal feature representations to build strong classifiers using Convolutional Neural Networks. We link our predictions to produce detections consistent in time, which we call action tubes. We show that our approach outperforms other techniques in the task of action detection.

In this paper, we are interested in the object detection problem with high accuracy and fast prediction. To achieve this, we have modified the existing object detection algorithm, we design a new feature extraction network, predict detection target based on multiple feature maps and improve the anchor box. Particularly, since DenseNet can combine the output of previous convolutional layers by using skip connection operation to make prediction more accurate, we introduce DenseNet to improve the YOLOv3-Tiny network. Furthermore, we add a prediction layer to the improved network to detect more true targets. Finally, we clustered the ground truth of specific dataset to get the anchor box for training to improve the detection results of the improved network. We demonstrate the effectiveness of our network by using Pascal VOC datasets. The experimental results show that the improved network increases the detection accuracy by 14% and the detection speed is 63 frames per second in the video detection. The detection accuracy is improved to a higher level and the detection speed still meeting the real-time detection requirement, which shows that the proposed method is robust and effective. Anchor box, DenseNet, Video detection

Video object detection is essential for airport surface surveillance, but the objects on the scene are mostly small objects with low resolution, they have no obvious feature information. Due to the scale differences of the objects and the fixed receptive field on the feature maps, detectors cannot model multi‐scale context information and cover all objects. In addition, although the video detection algorithm can be used as a method to solve the problem of small object detection, the temporal feature fusion method of current video detection is very dependent on the quality of a single feature map. Therefore, this paper aims to enhance the features of small objects of a single image. First, an attentional multi‐scale feature fusion enhancement (A‐MSFFE) network is built on the memory‐enhanced global‐local aggregation (MEGA) to supplement semantic and spatial information of small objects. Then, a context feature enhancement (CFE) module is designed for obtaining different receptive fields through different dilated convolutions. Meanwhile, a video detection dataset about the airport is established. Finally, the experimental results show that the proposed method can improve the detection accuracies of small objects and outperform other state‐of‐the‐art video object detection algorithms in self‐built airport dataset.

During past few years’ significant usage of video monitoring has been observed in several places. Video monitoring has always helped in improvement of security specific aspects. These systems provide major helping hand in monitoring through multiple object detection methods. However, there are certain challenges related to weather conditions which needs to be addressed. In view of this in this paper, we study multiple object detection in surveillance videos through hybrid deep gaussian mixture ensembles. This method integrates multiple deep gaussian mixture components to form ensembles which helps to develop multiple object detection solutions in surveillance videos. The computational system consists of ensemble of three major deep gaussian mixture components. The ensemble pipeline is formed with augmenting, bagging and stacking methods. The ensemble consists of hybrid object detection paradigm with steps background analysis, video pre-processing, data integration, object detection, foreground detection and post-processing. The computational pipeline is successfully experimented with ViSOR and CDnet 2014 benchmarked datasets. All results are validated with accuracy, precision, sensitivity or recall, specificity, F1-Score and RMSE metrics. Several comparative studies are performed with state-of-the-art methods as well as baselines. The experimental results demonstrate superiority of this method in comparison with other methods. Ablation studies have also been performed here with superior results. This system presents robustness in multiple object detection for real life challenges such as sudden illumination variation, shadow presence, long term occlusion and formation of dynamic backgrounds. It provides cost-effective, more profitable, efficient and sustainable real time multiple object detection solutions

Automated event recognition in video data has numerous practical applications for security and transportation. The ability to recognize events in practice depends on precisely detecting and tracking objects of interest in the video data. Numerous factors, such as lighting, weather, camera placement, scene complexity, and data compression can degrade the performance of automated algorithms. As a preprocessing step, developing a set of robust background models can substantially improve system performance. Our object detection and tracking algorithms estimate the object position and attributes within the context of this model to provide more reliable event recognition under challenging conditions. We present an approach to robustly modeling the background as a function of the data acquisition conditions. One element of this approach is automated assessment of the image quality which informs the choice of which background model to use for a given video stream. The video quality model rests on a suite of image metrics computed in real-time from the video, whereas the background models are constructed from historical data collected over a range of conditions. We will describe the formulation of both models. Results from a recent experiment will quantify the empirical performance for recognition of events of interest.

The dominant paradigm in spatiotemporal action detection is to classify actions using spatiotemporal features learned by 2D or 3D Convolutional Networks. We argue that several actions are characterized by their context, such as relevant objects and actors present in the video. To this end, we introduce an architecture based on self-attention and Graph Convolutional Networks in order to model contextual cues, such as actor-actor and actor-object interactions, to improve human action detection in video. We are interested in achieving this in a weakly-supervised setting, i.e. using as less annotations as possible in terms of action bounding boxes. Our model aids explainability by visualizing the learned context as an attention map, even for actions and objects unseen during training. We evaluate how well our model highlights the relevant context by introducing a quantitative metric based on recall of objects retrieved by attention maps. Our model relies on a 3D convolutional RGB stream, and does not require expensive optical flow computation. We evaluate our models on the DALY dataset, which consists of human-object interaction actions. Experimental results show that our contextualized approach outperforms a baseline action detection approach by more than 2 points in Video-mAP. Code is available at https://github.com/micts/acgcn.

We present a 3D Convolutional Neural Networks (CNNs) based single shot detector for spatial-temporal action detection tasks. Our model includes: (i) two short-term appearance and motion streams, with single RGB and optical flow image input separately, in order to capture the spatial and temporal information for the current frame; (ii) two long-term 3D ConvNet based stream, working on sequences of continuous RGB and optical flow images to capture the context from past frames. Our model achieves strong performance for action detection in video and can be easily integrated into any current two-stream action detection methods. We report a frame-mAP of 71.30% on the challenging UCF101-24 [1] actions dataset, achieving the state-of-the-art result of the one-stage methods. To the best of our knowledge, our work is the first system that combined 3D CNN and SSD in action detection tasks.

Head detection in real-world videos is a classical research problem in computer vision. Head detection in videos is challenging than in a single image due to many nuisances that are commonly observed in natural videos, including arbitrary poses, appearances, and scales. Generally, head detection is treated as a particular case of object detection in a single image. However, the performance of object detectors deteriorates in unconstrained videos. In this paper, we propose a temporal consistency model (TCM) to enhance the performance of a generic object detector by integrating spatial-temporal information that exists among subsequent frames of a particular video. Generally, our model takes detection from a generic detector as input and improves mean average precision (mAP) by recovering missed detection and suppressing false positives. We compare and evaluate the proposed framework on four challenging datasets, i.e., HollywoodHeads, Casablanca, BOSS, and PAMELA. Experimental evaluation shows that the performance is improved by employing the proposed TCM model. We demonstrate both qualitatively and quantitatively that our proposed framework obtains significant improvements over other methods.

In this paper, we investigate the hardware accel-eration of the forward and backward passes with quantized arithmetic in graph convolutional (GCN) and attention networks (GAT). The proposed architecture uses adaptive fixed-point numeric precisions down to 1-bit in the forward pass and 4-bit in the backward pass. The accelerator is mapped to a Zynq UltraScale+ MPSOC and integrated within the PyTorch framework, delivering scalable performance with multiple configuration options that include varying the number of hardware threads and compute units available in each thread. Each hard-ware thread uses a hierarchical dataflow with fine and coarse-grained components. The coarse-grained dataflow is composed of aggregation, combination, and attention stages. Hardware-aware quantization is used to optimize the parameter width of weights, features, attention coefficients, and graph connectivity expressed in the normalized adjacency matrix. Performance evaluation using popular Planetoid graph datasets shows acceleration factors higher than 2 orders of magnitude for GCN and GAT compared with on-device CPU execution. The experimental measurements compared with other FPGA, mobile GPU, and CPU hardware show up to 50x better performance per Joule.

Hybrid bee colony optimization for examination timetabling problems

Object detection in videos is widely used in entertainment, robotics, surveillance etc. A major challenge in effective object detection is observed when there is occlusion, bad illumination or cluttered background. Besides, cameras do not provide any mechanism for detecting moving objects after capture. Researchers have proposed different methods for object detection in video frames, ranging from traditional to deep learning approaches. However, using the right method in the right situation for efficient and accurate detection is a concern. Even though deep learning has shown high accuracy in detection, the training and testing time and cost is a concern. In our research, we explore the best motion models for tracking objects in a video sequence. The characteristics of each model was analyzed and the results were compared using different types of videos. Our observations determine that depending on the quality of input video, traditional approaches show high accuracy in detection, comparable to state-of-the art methods.

Since CNN began airing its programmes, the interdisciplinary subject of computer vision has experienced tremendous growth, and self-driving cars have assumed centre stage. Object detection is a crucial component of computer vision. Object detection helps with surveillance, vehicle detection, pose estimation, and other tasks. A simplistic solution to this issue would be to select distinct areas of interest from the image and use a CNN to determine whether the object is present in that area. The issue with this method is that the objects of interest could be located differently in the image's space and have various aspect ratios. Therefore, you would have to choose a big number of locations, which could cause a computational disaster. Recent studies on image recognition tasks have demonstrated the superior performance of Recurrent Convolutional Neural Networks (RCNNs). In this study, we take it a step further and discuss the challenge of object detection using DNNs, i.e., accurately localizing objects of different classes in addition to classifying them. We describe an approach to object detection as a regression problem to object bounding box masks that is both straightforward and effective. We outline a multi-scale inference method that a few application programs can use to quickly and affordably detect objects at high resolution. On several datasets, the technique performs at the cutting edge.

This paper addresses the challenging problem of generating videos with human action semantics. Unlike previous work which predict future frames in a single forward pass, this paper introduces the cycle constraints in both forward and backward passes in the generation of human actions. This is achieved by enforcing the appearance and motion consistency across a sequence of frames generated in the future. The approach consists of two stages. In the first stage, the pose of a human body is generated. In the second stage, an image generator is used to generate future frames by using (a) generated human poses in the future from the first stage, (b) the single observed human pose, and (c) the single corresponding future frame. The experiments are performed on three datasets: Weizmann dataset involving simple human actions, Penn Action dataset and UCF-101 dataset containing complicated human actions, especially in sports. The results from these experiments demonstrate the effectiveness of the proposed approach.

This paper proposes a data augmentation strategy for improving on-road object detection based on a deep neural network. The method uses a single camera and detects objects based on an optimized deep neural network for a driving environment. The strategy also uses a single-shot multi-box detector (SSD) for object detection, which is a state-of-the-art deep-learning algorithm. The performance is improved by using data augmentation for an advanced driver assist system (ADAS) specific to on-road object recognition. The problem of object detection is first analyzed based on a deep neural network in the ADAS domain, and then representative object detection methods that use deep neural networks are surveyed. A restricted random crop process is suggested for detecting small objects in an image, and then a patch resampling strategy is proposed for solving the long tail property in an on-road dataset. The proposed ADAS domain-specific data augmentation method is adjusted for the original object detection method based on a deep neural network. The object detection results were evaluated using an embedded board on the KITTI benchmark dataset, and the suggested data augmentation method improves the average precision by 30%.