NIC: A Robust Background Extraction Algorithm for Foreground Detection in Dynamic Scenes

A method is presented for extracting object information from an image sequence taken by a static monocular camera. The method was developed towards a low computational complexity in order to be used in real-time surveillance applications. Our approach makes use of both intensity and edge information of each frame and works efficiently in an indoor environment. It consists of two major parts: background processing and foreground extraction. The background estimation and updating makes the object detection robust to environment changes like illumination changes and camera jitter. The fusion of intensity and edge information allows a more precise estimation of the position of the different foreground objects in a video sequence. The result obtained are quite reliable, under a variety of environmental conditions.

Camera jitter occurs frequently in outdoor scenes and poses a great challenge to foreground detection. To meet this challenge, we propose a background subtraction method based on online robust principal component analysis (OR-PCA). We downsample every input frame in a random manner to make the OR-PCA applicable to video processing. Different from most background subtraction methods which rely on pixel-based background models, our method utilizes the low-dimensional subspace constructed by backgrounds of previous frames to estimate background of a new input frame, and hence can well handle the camera jitter. We find that the resulting sparse matrix contains not only the foreground objects but also some sparse noise, and then eliminate the sparse noise to improve the precision of our method. Experimental results demonstrate that, our method achieves remarkable results and outperforms several advanced methods in dealing with the camera jitter.

Object detection and tracking under complex environment are challenging because of the disturbances induced by background clutter, illumination changes, occlusions and other factors. The bulk of traditional algorithms basically rely on hand‐crafted features, which are not sufficiently robust to a complex environment. Moreover, the processes of detection and tracking are separated, which leads to the overall efficiency not high. In this study, a novel local probability model (LPM)‐based mean shift (MS) algorithm is proposed to integrate object detection and tracking. The main contributions include: (i) a new framework based on the combination of LPM and MS is established for the integration of object tracking and detection. (ii) For object detection, the training and prediction of LPM are built by stacked denoising autoencoders based deep learning. (iii) For object tracking, an MS tracking algorithm leveraging LPM is modified to improve the tracking efficiency under a complex environment. Experimental results demonstrate that the proposed method is superior to the colour histograms based MS and histograms of oriented gradients based MS in terms of robustness and tracking accuracy.

Background subtraction (BS) is a popular method for detecting moving objects from static cameras able to achieve real-time performance. BS aims to identify moving regions in image sequences comparing the current frame to a model of the scene background (BG). The creation of such a model is a challenging task due to illumination changes (gradual and sudden), shadows, camera jitter, movement of background elements (e.g., trees swaying in the breeze, waves in water), and changes in the background geometry (e.g., parked cars).

A consensus-based method for tracking: Modelling background scenario and foreground appearance

One of the fundamental operations in computer vision applications is change detection, in which moving foreground objects are segmented from a static background. A common approach for change detection is the comparison of an image frame with the stored background model using a matching algorithm, a process known as background subtraction. However, such techniques fail in environments with dynamic backgrounds, illumination changes, or shadow and camera jitters. This study focuses on effectively detecting scene changes in complex environments. To this end, we proposed a new colour descriptor named local colour difference pattern (LCDP) that is insusceptible to shadow and is able to capture both colour and texture features at a pixel location. Furthermore, a scene change detection framework was proposed to handle dynamic scenes based on sample consensus that integrates LCDP and a novel spatial model fusion mechanism. Experiments using the CDnet benchmark dataset demonstrated the effectiveness of the proposed approach to change detection in complex environments.

Abstract. The automatic analysis of 3D point clouds has become a crucial task in photogrammetry, remote sensing and computer vision. Whereas modern range cameras simultaneously provide both range and intensity images with high frame rates, other devices can be used to obtain further information which could be quite valuable for tasks such as object detection or scene interpretation. In particular thermal information offers many advantages, since people can easily be detected as heat sources in typical indoor or outdoor environments and, furthermore, a variety of concealed objects such as heating pipes as well as structural properties such as defects in isolation may be observed. In this paper, we focus on thermal 3D mapping which allows to observe the evolution of a dynamic 3D scene over time. We present a fully automatic methodology consisting of four successive steps: (i) a radiometric correction, (ii) a geometric calibration, (iii) a robust approach for detecting reliable feature correspondences and (iv) a co-registration of 3D point cloud data and thermal information via a RANSAC-based EPnP scheme. For an indoor scene, we demonstrate that our methodology outperforms other recent approaches in terms of both accuracy and applicability. We additionally show that efficient straightforward techniques allow a categorization according to background, people, passive scene manipulation and active scene manipulation.

Detection of moving objects is a fundamental task in video based surveillance and security applications. Many detection systems use background estimation methods to model the observed environment. In outdoor surveillance, moving backgrounds (waving trees, clutter) and illumination changes (weather changes, reflections, etc.) are the major challenges for background modelling and the development of a single model that fulfils all these requirements is usually not possible. In this paper we present a background estimation technique for motion detection in non-static backgrounds that overcomes this problem. We introduce an enhanced background estimation architecture with a long-term model and a short-term model. Our system showed that fusion of the detections of these two complementary approaches, improves the quality and reliability of the detection results.

Moving Object Detection (MOD) is one of the prominent application in the field of computer vision. This paper contributes towards designing a fast and accurate MOD technique for real time applications by addressing the issues like dynamic background, illumination changes, occlusion, camera jitter, moving shadows, etc. The brilliance of the proposed method is confined in the effective utilization of QR decomposition and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$l_{2,1}$</tex> -norm minimization in the well-known Robust Principal Component Analysis (RPCA) framework. The <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$l_{2,1}$</tex> norm is robust to outliers and exploits the low rankness of the background component. While the use of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$l_{1}$</tex> -norm enhances the sparsity in the foreground part. Moreover, Total Variation(TV) regularization technique helps in retaining the details about the edges. Experimental result shows the supremacy of the proposed method w.r.t. the state of the art methods in terms of computational complexity and F-measure score.

Local change detection for moving object detection is an essential step in any computer vision task. The most well-known technique is background subtraction BGS. However, the performance of BGS is strongly dependent on the background construction. The background construction to be robust in the presence of various challenges: dynamic backgrounds, illumination changes, camera jitter, etc. In this paper, we propose a novel encoder-decoder-based end-to-end deep learning framework for BGS. Thus, we explore a VGG-19 deep network with a transfer learning strategy as an encoder that deeply learned and extracted the features at different levels. We herewith propose a Multi-scale Feature Pulling MFP block which can retain the features at the various scales of the challenging video scenes. We also design a decoder network which is a stack of several transposed convolutional layers which precisely predict that each pixel of the target frame belongs to the background or foreground. The efficiency of the proposed algorithm is validated on the CDNet-2014 dataset by comparing its results against seventeen state-of-the-art techniques.

The Golgi apparatus and membrane tubules derived from this organelle play essential roles in membrane trafficking in eukaryotic cells. High-resolution live cell imaging is one highly suitable method for studying the molecular mechanisms of dynamics of organelles during membrane trafficking events. Due to the complex morphological changes and dynamic movements of the Golgi apparatus and associated membrane tubules during membrane trafficking, it is challenging to accurately quantify them. In this study, a semi-automated 2D tracking system, 2D-GolgiTrack, has been established for quantifying morphological changes and movements of Golgi elements, specifically encompassing the Golgi apparatus and its associated tubules, the fission and fusion of Golgi tubules, and the kinetics of formation of Golgi tubules and redistribution of the Golgi-associated protein Rab6A to the endoplasmic reticulum. The Golgi apparatus and associated tubules are segmented by a combination of Otsu's method and adaptive local normalization thresholding. Curvilinear skeletons and tips of skeletons of segmented tubules are used for calculating tubule length by the Geodesic method. The k-nearest neighbor is applied to search the possible candidate objects in the next frame and link the correct objects of adjacent frames by a tracking algorithm to calculate changes in morphological features of each Golgi object or tubule, e.g., number, length, shape, branch point and position, and fission or fusion events. Tracked objects are classified into morphological subtypes, and the Track-Map function of morphological evolution visualizes events of fission and fusion. Our 2D-GolgiTrack not only provides tracking results with 95% accuracy, but also maps morphological evolution for fast visual interpretation of the fission and fusion events. Our tracking system is able to characterize key morphological and dynamic features of the Golgi apparatus and associated tubules, enabling biologists to gain a greater understanding of the molecular mechanisms of membrane traffic involving this essential organelle. Graphical Abstract Overview of the semi-automated 2D tracking system. There are two main parts to the system, namely detection and tracking. The workflow process requires a raw sequence of images (a), which is filtered by the Gaussian filter method (c), and threshold intensity (b) to segment elements of Golgi cisternae (d) and tubules (e). Post-processing outputs are binary images of the cisternae area and tubule skeletons. The tubules are classified into three lengths, namely short, medium, and long tubules (f). Outputs of segmentation are calculated as morphological features (g). The tracking processing starts by loading the segmented outputs (h) and key-inputs of direction reference (i; (DR)) and interval setting of the start ((S)) and end ((E)) frame numbers (j). A tubule of interest is selected by the user (k; (GTinterest, S) as the tubule input ((GTIN)) at the current frame ((i = S)). The tracking algorithm tracks and links the correct tubules at each subsequent frame ((i = i + 1)). The locations of tubule tips are determined for detecting tubule branches using the (DR) to identify the direction of tubule growth (l: (1); (GTtipBr, i); Golgi cisternae: white area; Golgi tubule: white skeleton; tubule tips: green dots; branched tubules: two branches due to the (DR) of growth of the simulated tubule moving from left-to-right away from the Golgi cisternae location). According to the position of the (GTIN), five candidates ((GTcandidates, i)) are searched using the k-nearest neighbor method (l: (2)). Matching of tubules between the (GTIN) and those (GTcandidates, i) uses the bounding box technique to check the amount of tubule-overlap based on the tracking conditions (l: (3)). If there is tubule-overlap, the system collects that tubule as the final output ((GTOUT, i)). By contrast, shape (see the Extent feature in Table reftab:1) and distance features are used to generate the tracked output, which has a priority of a minimum of both of these features ((MinDIST, EXTENT)); otherwise, it is from the minimum of the distance ((MinDIST)). Once a loop of the interval track to the last frame is finished ((i = E + 1)), a Track-Map is generated allowing visualization of the morphological pattern of tubule formation and movement, including identification of fission and fusion events (m). Dynamic features are calculated (n). Related outputs are saved, and all features obtained from the detection and tracking processing are exported as MS Excel files (o).

Automated traffic surveillance systems are widely used in intelligent transportation systems (ITS). However, the accuracy of video-based Vehicle Detection is heavily affected by complex environmental factors such as shadows, rain, illumination and glare. This paper introduces an approach to motion vehicle detection in complex condition over highway surveillance video. The framework is composed of two parts: background estimation and multi-feature extraction. A fast constrained Delaunay triangulation (CDT) algorithm based on constrained-edge priority is presented instead of complicated segmentation algorithms. We present a background block update modeling theory based on triangulation to estimate background self-adaptively. Consequently, we can get the difference between the current frame and the background model. After extracting features in triangular candidates, multi-feature eigenvector is created for each vehicle with Principal Component Analysis (PCA). We design a classifier to classify triangular candidate as a part of a real vehicle or not by Support Vector Machine (SVM). And then, a parallelogram is used to represent the vehicle's shape robustly. Finally, experiments using real video sequence are performed to verify the method proposed for complex environmental factors.

This paper presents a fast method for detecting textured objects moving in a video sequence. It is based on a known background estimation and a fixed camera position. The algorithm is able to detect the presence of moving objects and locates them on-line. It is a frame-by-frame method. First, a difference image is computed from the background and the current frame. This yields three classes of pixels, those for which something changed with respect to the background, those for which nothing changed, and finally the pixels for which no decision can be made. Then an a contrario model allows an automatic clustering, by using adapted rectangular windows, of the pixels for which changes have been detected. If necessary, these regions are corrected in order to better fit the moving objects' boundaries. Experimental results show that the algorithm is very robust with respect to noise and to the quality of the background estimation. The choice of the model parameters is quite natural and user friendly. The algorithm has been successfully tested on video sequences coming from different databases, including indoor and outdoor sequences.

The video-based separation of foreground (FG) and background (BG) has been widely studied due to its vital role in many applications, including intelligent transportation and video surveillance. Most of the existing algorithms are based on traditional computer vision techniques that perform pixel-level processing assuming that FG and BG possess distinct visual characteristics. Recently, state-of-the-art solutions exploit deep learning models targeted originally for image classification. Major drawbacks of such a strategy are the lacking delineation of FG regions due to missing temporal information as they segment the FG based on a single frame object detection strategy. To grapple with this issue, we excogitate a 3D convolutional neural network (3D CNN) with long short-term memory (LSTM) pipelines that harness seminal ideas, viz., fully convolutional networking, 3D transpose convolution, and residual feature flows. Thence, an FG-BG segmenter is implemented in an encoder-decoder fashion and trained on representative FG-BG segments. The model devises a strategy called double encoding and slow decoding, which fuses the learned spatio-temporal cues with appropriate feature maps both in the down-sampling and up-sampling paths for achieving well generalized FG object representation. Finally, from the Sigmoid confidence map generated by the 3D CNN-LSTM model, the FG is identified automatically by using Nobuyuki Otsu's method and an empirical global threshold. The analysis of experimental results via standard quantitative metrics on 16 benchmark datasets including both indoor and outdoor scenes validates that the proposed 3D CNN-LSTM achieves competitive performance in terms of figure of merit evaluated against prior and state-of-the-art methods. Besides, a failure analysis is conducted on 20 video sequences from the DAVIS 2016 dataset.

Change detection with background subtraction process remains to be an unresolved issue and attracts research interest due to challenges encountered on static and dynamic scenes. The key challenge is about how to update dynamically changing backgrounds from frames with an adaptive and self-regulated feedback mechanism. In order to achieve this, we present an effective change detection algorithm for pixelwise changes. A sliding window approach combined with dynamic control of update parameters is introduced for updating background frames, which we called sliding window-based change detection. Comprehensive experiments on related test videos show that the integrated algorithm yields good objective and subjective performance by overcoming illumination variations, camera jitters, and intermittent object motions. It is argued that the obtained method makes a fair alternative in most types of foreground extraction scenarios; unlike case-specific methods, which normally fail for their nonconsidered scenarios.