Pixel-to-Model Distance for Robust Background Reconstruction

On the role and the importance of features for background modeling and foreground detection

Detection of video objects under bad weather and poor illumination condition is a challenging task. We address this issue using the notion of background modeling. LBP-based background modeling and model learning has been used to detect the video objects but performance degrades in the above complex background. We propose the adjacency- and reinforced adjacency-based variants of LBP for complex real-world background modeling and model learning for object detection. In this regard, we have proposed the four variants; (1) enhanced adjacent local binary pattern, (2) enhanced reinforced adjacent local binary pattern, (3) enhanced adjacent local ternary pattern, and (4) enhanced reinforced adjacent local ternary pattern. Besides, we have embedded the Gabor and LBP features to obtain an embedded feature, which is subsequently used with the notions of adjacency. Unlike the background learning approach where the model learns only the background, our model learning algorithm learns the background together with the foreground objects and hence named as mixed-mode learning strategy. These models together with the new learning strategy are tested with CD2014 (snowfall and blizzard) and PETS (2014 and 2016) data sets, and the performance of the proposed models has been compared with LBP-based methods and other state-of-the-art methods.

It is well known that frontal face is much easier to be recognized than pose-variant face for both human and machine perception. However, it is not easy to acquire a frontal face in real-world video surveillance. This paper proposes a method to synthetize a frontal face for recognition in video surveillance scene, which is based on Conditional Generative Adversarial Networks (cGAN) with input of multiple pose-variant faces from a video. Experimental results show that the proposed approach can generate suitable frontal faces and improve face recognition by around 20% on a dataset of 43276 face images from 19 persons, collected from the real-world video surveillance scene. The effectiveness of multiple frames against single frame as input is demonstrated. Moreover, we investigate the generator with different depth for synthetizing frontal faces, in which an up-down sampling trick is designed for synthetizing higher quality frontal face images and boosts the performance of the generator.

Automated event recognition in video data has numerous practical applications for security and transportation. The ability to recognize events in practice depends on precisely detecting and tracking objects of interest in the video data. Numerous factors, such as lighting, weather, camera placement, scene complexity, and data compression can degrade the performance of automated algorithms. As a preprocessing step, developing a set of robust background models can substantially improve system performance. Our object detection and tracking algorithms estimate the object position and attributes within the context of this model to provide more reliable event recognition under challenging conditions. We present an approach to robustly modeling the background as a function of the data acquisition conditions. One element of this approach is automated assessment of the image quality which informs the choice of which background model to use for a given video stream. The video quality model rests on a suite of image metrics computed in real-time from the video, whereas the background models are constructed from historical data collected over a range of conditions. We will describe the formulation of both models. Results from a recent experiment will quantify the empirical performance for recognition of events of interest.

Recent advances in the field of object detection and face recognition have made it possible to develop practical video surveillance systems with embedded object detection and face recognition functionalities that are accurate and fast enough for commercial uses. In this paper, we compare some of the latest approaches to object detection and face recognition and provide reasons why they may or may not be amongst the best to be used in video surveillance applications in terms of both accuracy and speed. It is discovered that Faster R-CNN with Inception ResNet V2 is able to achieve some of the best accuracies while maintaining real-time rates. Single Shot Detector (SSD) with MobileNet, on the other hand, is incredibly fast and still accurate enough for most applications. As for face recognition, FaceNet with Multi-task Cascaded Convolutional Networks (MTCNN) achieves higher accuracy than advances such as DeepFace and DeepID2+ while being faster. An end-to-end video surveillance system is also proposed which could be used as a starting point for more complex systems. Various experiments have also been attempted on trained models with observations explained in detail. We finish by discussing video object detection and video salient object detection approaches which could potentially be used as future improvements to the proposed system.

Object detection is a hot topic with various applications in computer vision, e.g., image understanding, autonomous driving, and video surveillance. Much of the progresses have been driven by the availability of object detection benchmark datasets, including PASCAL VOC, ImageNet, and MS COCO. However, object detection on the drone platform is still a challenging task, due to various factors such as view point change, occlusion, and scales. To narrow the gap between current object detection performance and the real-world requirements, we organized the Vision Meets Drone (VisDrone2018) Object Detection in Image challenge in conjunction with the 15th European Conference on Computer Vision (ECCV 2018). Specifically, we release a large-scale drone-based dataset, including 8, 599 images (6, 471 for training, 548 for validation, and 1, 580 for testing) with rich annotations, including object bounding boxes, object categories, occlusion, truncation ratios, etc. Featuring a diverse real-world scenarios, the dataset was collected using various drone models, in different scenarios (across 14 different cities spanned over thousands of kilometres), and under various weather and lighting conditions. We mainly focus on ten object categories in object detection, i.e., pedestrian, person, car, van, bus, truck, motor, bicycle, awning-tricycle, and tricycle. Some rarely occurring special vehicles (e.g., machineshop truck, forklift truck, and tanker) are ignored in evaluation. The dataset is extremely challenging due to various factors, including large scale and pose variations, occlusion, and clutter background. We present the evaluation protocol of the VisDrone-DET2018 challenge and the comparison results of 38 detectors on the released dataset, which are publicly available on the challenge website: http://www.aiskyeye.com/. We expect the challenge to largely boost the research and development in object detection in images on drone platforms.

Background models have been widely used for video surveillance and other applications. Methods for constructing background models and associated application algorithms are mainly studied in spatial domain (pixel level). Many video sources, however, are in a compressed format before processing. In this paper, we propose an approach to construct background models directly from compressed video data. The proposed approach utilizes the information exploited from DCT coefficients at block level to construct accurate background models at pixel level. We implemented three representative algorithms of background models in the compressed domain, and theoretically explored their properties and the relationship with their counterparts in the spatial domain. We also present some general technical improvements to make the proposed approach more capable for a wide range of applications. The proposed method can achieve the same accuracy as the methods that construct background models from the spatial domain with much lower computational cost (26 % on average) and more compact storages. 1.

Video surveillance is one of the most active research topics in the computer vision due to the increasing need for security. Although surveillance systems are getting cheaper, the cost of having human operators to monitor the video feed can be very expensive and inefficient. To overcome this problem, the automated visual surveillance system can be used to detect any suspicious activities that require immediate action. The framework of a video surveillance system encompasses a large scope in machine vision, they are background modelling, object detection, moving objects classification, tracking, motion analysis, and require fusion of information from the camera networks. This paper reviews recent techniques used by researchers for detection of moving object detection and tracking in order to solve many surveillance problems. The features and algorithms used for modelling the object appearance and tracking multiple objects in outdoor and indoor environment are also reviewed in this paper. This paper summarizes the recent works done by previous researchers in moving objects tracking for single camera view and multiple cameras views. Nevertheless, despite of the recent progress in surveillance technologies, there still are challenges that need to be solved before the system can come out with a reliable automated video surveillance.

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT REQUEST OF AUTHOR.] Camera traps are a popular tool to sample animal populations because they are noninvasive, detect a variety of species, and can record many thousands of animal detections per deployment. Cameras are typically set to take bursts of multiple images for each detection, and are deployed in arrays of dozens or hundreds of sites, often resulting in millions of images per study. The task of converting images to animal detection records from such large image collections is daunting, and made worse by situations that generate copious empty pictures from false triggers (e.g. camera malfunction or moving vegetation) or pictures of humans. We offer the first widely available computer vision tool for processing camera trap images. Our results show that the tool is accurate and results in substantial time savings for processing large image datasets, thus improving our ability to monitor wildlife across large scales with camera traps. In this dissertation, we have developed new image/video processing and computer vision algorithms for efficient and accurate object detection and sequence-level classiffication from natural scene camera-trap images. This work addresses the following five major tasks: (1) Human-animal detection. We develop a fast and accurate scheme for human-animal detection from highly cluttered camera-trap images using joint background modeling and deep learning classification. Specifically, first, We develop an effective background modeling and subtraction scheme to generate region proposals for the foreground objects. We then develop a cross-frame image patch verification to reduce the number of foreground object proposals. Finally, We perform complexity-accuracy analysis of deep convolutional neural networks (DCNN) to develop a fast deep learning classification scheme to classify these region proposals into three categories: human, animals, and background patches. The optimized DCNN is able to maintain high level of accuracy while reducing the computational complexity by 14 times. Our experimental results demonstrate that the proposed method outperforms existing methods on the camera-trap dataset. (2) Object segmentation from natural scene. We first design and train a fast DCNN for animal-human-background object classification, which is used to analyze the input image to generate multi-layer feature maps, representing the responses of different image regions to the animal-human-background classifier. From these feature maps, we construct the so-called deep objectness graph for accurate animal-human object segmentation with graph cut. The segmented object regions from each image in the sequence are then verfied and fused in the temporal domain using background modeling. Our experimental results demonstrate that our proposed method outperforms existing state-of-the-art methods on the camera-trap dataset with highly cluttered natural scenes. (3) DCNN domain background modeling. We replaced the background model with a new more efficient deep learning based model. The input frames are segmented into regions through the deep objectness graph then the region boundaries of the input frames are multiplied by each other to obtain the regions of movement patches. We construct the background representation using the temporal information of the co-located patches. We propose to fuse the subtraction and foreground/background pixel classiffcation of two representation : a) chromaticity and b) deep pixel information. (4) Sequence-level object classiffcation. We proposed a new method for sequence-level video recognition with application to animal species recognition from camera trap images. First, using background modeling and cross-frame patch verification, we developed a scheme to generate candidate object regions or object proposals in the spatiotemporal domain. Second, we develop a dynamic programming optimization approach to identify the best temporal subset of object proposals. Third, we aggregate and fuse the features of these selected object proposals for efficient sequence-level animal species classification.

Background subtraction method is widely used in most of the video motion detection algorithms especially for video surveillance application. Background subtraction is used to extract moving or static foreground objects from the background scene. The efficiency of foreground-background segmentation heavily relies on background model which must be able to cope with changes in the scene and granularity of the foreground objects. The robust background model can produce good foreground segmentation results and it is still a great challenge to get accurate and high performance result today. In this paper, a video foreground-background segmentation approach is proposed. This approach is based on Codebook (CB) model with Kalman Filter. This approach can be used to extract foreground objects from the video stream. The Lab color space is used in this approach to calculate color difference between two pixels using CIEDE2000 color difference formula. Extracted foreground object from video sequence using this approach is useful for object detection in video surveillance applications. This approach has been tested with PETS and CDnet2014 datasets and segmentation results accuracy are evaluated compare with ground truth.

In this paper, we propose a novel background subtraction method which enables reliable detection of foreground objects in a long surveillance video stream. Recently, although much progress has been made in the field of background subtraction, there are still challenging scenarios (e.g., high frequency motion of dynamic texture, non-stationary motion of camera, abrupt changes of illumination, etc.) in the long surveillance videos in which even state-of-the-art methods are often prone to fail. To cope with these challenging scenarios effectively, in the proposed method, a background model is initialized in a low-dimensional subspace and then updated periodically based on a novel recursive on-line ${(2{\rm D})}^2{\rm PCA}$ algorithm developed in this paper. Moreover, a threshold map is also updated in a scene-adaptive manner for labeling each pixel in a scene either foreground or background independently. Based on this on-line framework, the background of a surveillance video stream is reconstructed over time, thereby facilitating the detection of foreground objects reliably. In extensive experiments, we demonstrate that the proposed background subtraction method can cope with the aforementioned challenging scenarios more favorably than the state-of-the-art methods.

The problem of foreground detection in real‐time video surveillance applications is addressed. Proposes is a framework, which is computationally cheap and has low memory requirements. It combines two simple processing blocks, both of which are essentially background subtraction algorithms. The main novelty of the approach is a combination of an autoregressive moving average filter with two background models having different adaptation speeds. The first model, having a lower adaptation speed, models long‐term background and detects foreground objects by finding areas in the current frame which significantly differ from the proposed background model. The second model, with a higher adaptation speed, models the short‐term background and is responsible for finding regions in the scene with a high foreground object activity. The final foreground detection is built by combining the outputs from these building blocks. The foreground obtained by the long‐term modelling block is verified by the output of the short‐term modelling block, i.e. only the objects exhibiting significant motion are detected as real foreground objects. The proposed method results in a very good foreground detection performance at a low computational cost.

The world is marching towards digital communication and digital video surveillance is an indispensable part of it. Moving object detection and face detection play an important role in the video surveillance applications. It is a part of security oriented applications such as traffic systems, hospitals, banks. Background subtraction algorithm is used in order to detect the foreground. Due to the light intensity variations in dynamically changing environment, background subtraction module needs an adaptive threshold value for each input video frame. The pixel values above the threshold value are detected as foreground pixels. The range of adaptive threshold values usually varies from 10 to 85 in the proposed work. This gives a better detection capability in moving object detection wherein PSNR of 32.39 is obtained when two detected images with and without adaptive threshold are chosen as inputs to the PSNR block. Face detection is treated as an extension module of moving object detection wherein the input face image is subtracted from the test image with the help of adaptive threshold but finally the matching unit is the additional component which decides the match based on the global threshold value which is chosen as 183000 here. Here, instead of detecting the moving body, only face is considered for detection. Since real time videos need to be processed using FPGA kits, an area efficient moving object and face detection system is designed using Spartan 6 FPGA kit. The number of slice registers used has reduced from 365 to 320 in case of moving object detection, from 861 to 537 in case of face detection. The results obtained reveal that the use of adaptive threshold concept results in better detection process with PSNR of 32.39.

In recent years, video surveillance systems evolve a great interest as the application area. Recent studies have proven integration of digital image processing, computer vision, artificial intelligence and data analytics into video surveillance application. This paper provides a systematic review of the state-of-the-art video analysis techniques available in machine learning and deep learning. Video processing research trends illustrate a survey on practices like object detection, object recognition, object tracking, traffic control and monitoring, action and behaviour recognition, disaster management and so on. We further present focus on computational approaches for various challenges of video processing techniques, objectives of those techniques and evaluation based on multiple problems. In the end, we have discussed emerging issues in the field of video analytics and how machine learning and deep learning contribute to those issues.KeywordsVideo analysisObject detectionObject tracking and object classificationMachine learning and deep learning

Object detection has made impressive progress in recent years with the help of deep learning. However, state-of-the-art algorithms are both computation and memory intensive. Though many lightweight networks are developed for a trade-off between accuracy and efficiency, it is still a challenge to make it practical on an embedded device. In this paper, we present a system-level solution for efficient object detection on a heterogeneous embedded device. The detection network is quantized to low bits and allows efficient implementation with shift operators. In order to make the most of the benefits of low-bit quantization, we design a dedicated accelerator with programmable logic. Inside the accelerator, a hybrid dataflow is exploited according to the heterogeneous property of different convolutional layers. We adopt a straightforward but resource-friendly column-prior tiling strategy to map the computation-intensive convolutional layers to the accelerator that can support arbitrary feature size. Other operations can be performed on the low-power CPU cores, and the entire system is executed in a pipelined manner. As a case study, we evaluate our object detection system on a real-world surveillance video with input size of 512x512, and it turns out that the system can achieve an inference speed of 18 fps at the cost of 6.9W (with display) with an mAP of 66.4 verified on the PASCAL VOC 2012 dataset.