Detecting humans: analysis and synthesis as a pattern recognition problem

This work proposes a fast background learning algorithm for foreground detection under changing illumination. Gaussian Mixture Model (GMM) is an effective statistical model in background learning. We first focus on Titterington's online EM algorithm that can be used for real-time unsupervised GMM learning, and then advocate a deterministic data assignment strategy to avoid Bayesian computation. The color of the foreground is apt to be influenced by the environmental illumination that usually produce undesirable effect for GMM updating, however, a collinear feature of pixel intensity under changing light is discovered in RGB color space. This feature is afterward used as a reliable clue to decide which part of mixture to update under changing light. A foreground detection step proposed in early version of this work is employed to extract foreground objects by comparing the estimated background model with the current video frame. Experiments have shown the proposed method is able to achieve satisfactory static background images of scenes as well as is also superior to some mainstream methods in detection performance under both indoor and outdoor scenes.

The ray projection and its application to recovering a projective geometric transformation and an affine lighting change between two objects are mathematically studied. A novel technique, viz., variable contour, is proposed for accurately evaluating the ray projection. Moreover, a novel flexible framework of the ray projection is devised for the joint recovery of the eight parameters for the projective transformation and the two parameters for the lighting change. Finally, the framework is experimentally evaluated by recovering a variety of geometric and photometric transformations between real images of indoor and outdoor scenes. Its robustness to image blur and occlusion is demonstrated. Its versatility is illustrated through matching different objects in different classes.

Panoramic image differences can be used for view-based homing under natural outdoor conditions, because they increase smoothly with distance from a reference location (Zeil et al., J Opt Soc Am A 20(3):450-469, 2003). The particular shape, slope and depth of such image difference functions (IDFs) recorded at any one place, however, depend on a number of factors that so far have only been qualitatively identified. Here we show how the shape of difference functions depends on the depth structure and the contrast of natural scenes, by quantifying the depth- distribution of different outdoor scenes and by comparing it to the difference functions calculated with differently processed panoramic images, which were recorded at the same locations. We find (1) that IDFs and catchment areas become systematically wider as the average distance of objects increases, (2) that simple image processing operations -- like subtracting the local mean, difference-of-Gaussian filtering and local contrast normalization -- make difference functions robust against changes in illumination and the spurious effects of shadows, and (3) by comparing depth-dependent translational and depth-independent rotational difference functions, we show that IDFs of contrast-normalized snapshots are predominantly determined by the depth-structure and possibly also by occluding contours in a scene. We propose a model for the shape of IDFs as a tool for quantitative comparisons between the shapes of these functions in different scenes.

We develop a conceptually simple, flexible, and effective framework (named T-Net) for two-view correspondence learning. Given a set of putative correspondences, we reject outliers and regress the relative pose encoded by the essential matrix, by an end-to-end framework, which is consisted of two novel structures: "−" structure and "|" structure. " − " structure adopts an iterative strategy to learn correspondence features. "|" structure integrates all the features of the iterations and outputs the correspondence weight. In addition, we introduce Permutation-Equivariant Context Squeeze-and-Excitation module, an adapted version of SE module, to process sparse correspondences in a permutation-equivariant way and capture both global and channel-wise contextual information. Extensive experiments on outdoor and indoor scenes show that the proposed T-Net achieves state-of-the-art performance. On outdoor scenes (YFCC100M dataset), T-Net achieves an mAP of 52.28%, a 34.22% precision increase from the best-published result (38.95%). On indoor scenes (SUN3D dataset), T-Net (19.71%) obtains a 21.82% precision increase from the best-published result (16.18%). Source code: https://github.com/x-gb/T-Net.

Face perception is an essential and significant problem in pattern recognition, concretely including Face Recognition (FR), Facial Expression Recognition (FER), and Race Categorization (RC). Though handcrafted features perform well on face images, Deep Convolutional Neural Networks (DCNNs) have brought new vitality to this field recently. Vanilla DCNNs are powerful at learning high-level semantic features, but are weak in capturing low-level image characteristic changes in illumination, intensity, and texture regarded as key traits in facial processing and feature extraction, which is alternatively the strength of human-designed feature descriptors. To integrate the best of both worlds, we proposed novel Random Pixel Difference Convolution (RPDC) which is efficient alternatives to vanilla convolutional layers in standard CNNs and can promote to extract discriminative and diverse facial features. By means of searched RPDC of high efficiency, we build S-RaPiDiNet, and achieve promising and extensive experiment results in FR (<inline-formula> <tex-math notation="LaTeX">$\approx 0.5$ </tex-math></inline-formula>&#x0025; improvement), FER (over 1&#x0025; growth), and RC (0.25&#x0025;&#x2013;3&#x0025; increase) than baseline network in vanilla convolution, showing strong generalization of RPDC.

Template matching is a fundamental problem in pattern recognition, which has wide applications, especially in industrial inspection. In this paper, we propose a 1-D template matching algorithm which is an alternative for 2-D full search block matching algorithms. Our approach consists of three steps. In the first step the images are converted from 2-D into 1-D by summing up the intensity values of the image in two directions horizontal and vertical. In the second step, the template matching is performed among 1-D vectors using the similarity function sum of square difference. Finally, the decision will be taken based on the value of similarity function. Transformation template image and sub-images in the source image from 2-D grey level information into 1-D information vector reduce the dimensionality of the data and accelerate the computations. Experimental results show that the computational time of the proposed approach is faster and performance is better than three basic template matching methods. Moreover, our approach is robust to detect the target object with changes of illumination in the template also when the Gaussian noise added to the source image.

3D reconstruction of indoor scenes that contain non-rigidly moving human body using depth cameras is a task of extraordinary difficulty. Despite intensive efforts from the researchers in the 3D vision community, existing methods are still limited to reconstruct small scale scenes. This is because of the difficulty to track the camera motion when a target person moves in a totally different direction. Due to the narrow field of view (FoV) of consumer-grade red-green-blue-depth (RGB-D) cameras, a target person (generally put at about 2–3 meters from the camera) covers most of the FoV of the camera. Therefore, there are not enough features from the static background to track the motion of the camera. In this paper, we propose a system which reconstructs a moving human body and the background of an indoor scene using multiple depth cameras. Our system is composed of three Kinects that are approximately set in the same line and facing the same direction so that their FoV do not overlap (to avoid interference). Owing to this setup, we capture images of a person moving in a large scale indoor scene. The three Kinect cameras are calibrated with a robust method that uses three large non parallel planes. A moving person is detected by using human skeleton information, and is reconstructed separately from the static background. By separating the human body and the background, static 3D reconstruction can be adopted for the static background area while a method specialized for the human body area can be used to reconstruct the 3D model of the moving person. The experimental result shows the performance of proposed system for human body in a large-scale indoor scene.

Due to the rapid development of mobile devices, augmented reality (AR) has been utilized increasingly, and thus applications of AR have increased for indoor and outdoor scenes. However, most of them focus on dealing with indoor scenes and the research for outdoor scenes is rare. In this paper, we propose a system to explore an application of AR for presenting 3D architecture models in outdoor scenes. We aim to enhance the realism of a 3D architectural model for an outdoor scene and provide users with a realistic experience. We first capture a color image through a camera and detect a plane on the image. When obtaining the detected plane, we can generate an anchor and attach virtual objects (3D architecture models) to this anchor. After that, we use the environmental high dynamic range mode of ARCore to obtain lighting information. Thus, our system can present a more realistic light and dark contrast and shadows on the virtual objects. Moreover, we propose an occlusion handling technique to obtain the correct relative positions of real objects and virtual objects. We use a depth prediction model to obtain depth information enabling the real objects to occlude the virtual objects. Finally, the processed augmented color image is rendered. The experiments demonstrate that our system can produce favorable results within a reasonable distance.

Detection and tracking of moving objects is very important in various ways. Concerning the detection of moving objects by stationary cameras, the background looks different as the illumination changes. In this paper, we consider a particular image in an image sequence as the sum of a reference image containing the background and a difference image containing the moving objects but not the background. We show that a reference image and difference images can be obtained as the independent components of input images by Independent Component Analysis. Moving objects can then be located on the reference image and the difference images. Experimental results show that the proposed approach produces accurate detection of moving objects even if illumination changes.

International audience

In our work describes a method for accurately tracking persons in indoor surveillance video stream obtained from a static camera with difficult scene properties including illumination changes and solves the major occlusion problem by using Kalman filtering. First, moving objects are precisely extracted by determining its motion, for further processing. The scene illumination changes are averaged to obtain the accurate moving object during background subtraction process. In case of objects occlusion, we use the color feature information to accurately distinguish between objects. The method is able to identify moving persons, track them and provide unique tag for the tracked persons. The effectiveness of the proposed method is demonstrated with experiments in an indoor environment.

Background subtraction is widely used in detecting moving objects in a static scene, which requires a fixed camera, and a static background. Most background subtract algorithms require the illumination changing slowly, so can be tracked and updated. If the illumination changes fast and non-monotony, many algorithms fail. Here we research one special scene in changing illumination: single continuous varying illumination. This scene can be seen indoor or in the dark or in underground architecture and in some daytime conditions. We consider three different conditions on the background luminance, and a new algorithm based on these conditions through calculating the illumination factor is advanced, it can relative precisely obtain the moving objects in this scene compared to other algorithms.

One of the fundamental operations in computer vision applications is change detection, in which moving foreground objects are segmented from a static background. A common approach for change detection is the comparison of an image frame with the stored background model using a matching algorithm, a process known as background subtraction. However, such techniques fail in environments with dynamic backgrounds, illumination changes, or shadow and camera jitters. This study focuses on effectively detecting scene changes in complex environments. To this end, we proposed a new colour descriptor named local colour difference pattern (LCDP) that is insusceptible to shadow and is able to capture both colour and texture features at a pixel location. Furthermore, a scene change detection framework was proposed to handle dynamic scenes based on sample consensus that integrates LCDP and a novel spatial model fusion mechanism. Experiments using the CDnet benchmark dataset demonstrated the effectiveness of the proposed approach to change detection in complex environments.

Due to advancement in digital multimedia technologies, huge amount of video contents in such arenas as sports, surveillance, news, etc., is becoming widely available. Due to factors like presence of inactive frames, inter-view dependencies and frequent illumination changes, whole video content may not be important. Moreover, users may not have the adequate time to access/ store the entire video in real time. We propose a machine learning ensemble method to summarize the events in multiview videos, as a solution to the aforementioned issues. In order to make the accurate decision on keyframes, we trained our ensembles using meta approach where the inter-dependency and illumination changes of views are considered in the training phase. Experimental results on two benchmark datasets show that our model outperforms the state-of-the-art models with the best Recall and F-measure. The computing analysis of the model also indicates that it meets all requirements of real-time applications.

This paper presents a novel locality sensitive histogram algorithm for visual tracking. Unlike the conventional image histogram that counts the frequency of occurrences of each intensity value by adding ones to the corresponding bin, a locality sensitive histogram is computed at each pixel location and a floating-point value is added to the corresponding bin for each occurrence of an intensity value. The floating-point value declines exponentially with respect to the distance to the pixel location where the histogram is computed, thus every pixel is considered but those that are far away can be neglected due to the very small weights assigned. An efficient algorithm is proposed that enables the locality sensitive histograms to be computed in time linear in the image size and the number of bins. A robust tracking framework based on the locality sensitive histograms is proposed, which consists of two main components: a new feature for tracking that is robust to illumination changes and a novel multi-region tracking algorithm that runs in real time even with hundreds of regions. Extensive experiments demonstrate that the proposed tracking framework outperforms the state-of-the-art methods in challenging scenarios, especially when the illumination changes dramatically.