HUMAN POSTURE RECONSTRUCTION AND ANIMATION FROM MONOCULAR IMAGES BASED ON GENETIC ALGORITHMS

Recovering of the depth structure of a scene from monocular video content provides an important advantage in applications such as AR (placing and removing of objects) or 3D-TV and 3D cinema (2D-to-3D video conversion). In this paper, we present an automatic method to generate relative depth maps from monocular video sequences. It relies on the dynamic occlusion depth cue to recover the depth order of objects in the scene. The forward and backward motion analysis between each two consecutive frames allows the calculation of their dynamic occlusions. We estimate the motion using a modified version of the EpicFlow. Our modifications to this optical flow method made it coherent in forward-backward directions without compromising its performance. Thanks to this new feature, occlusions are simpler to calculate than the approaches used in the relevant literature. The obtained occlusions allow order deduction of the objects contained in the image. These objects are obtained using a segmentation approach which considers both color and motion. Ours results show a small improvement to the quality of the optical flow while adding the forward/backward coherence. With respect to the depth ordering our approach obtains slightly better results than the reference method while removing a computationally costly step from the processing.

This paper presents an approach to reconstruct non-stationary, articulated objects from silhouettes obtained with a monocular video sequence. We introduce the concept of motion blurred scene occupancies, a direct analogy of motion blurred images but in a 3D object scene occupancy space resulting from the motion/deformation of the object. Our approach starts with an image based fusion step that combines color and silhouette information from multiple views. To this end we propose to use a novel construct: the temporal occupancy point (TOP), which is the estimated 3D scene location of a silhouette pixel and contains information about duration of time it is occupied. Instead of explicitly computing the TOP in 3D space we directly obtain itpsilas imaged(projected) locations in each view. This enables us to handle monocular video and arbitrary camera motion in scenarios where complete camera calibration information may not be available. The result is a set of blurred scene occupancy images in the corresponding views, where the values at each pixel correspond to the fraction of total time duration that the pixel observed an occupied scene location. We then use a motion de-blurring approach to de-blur the occupancy images. The de-blurred occupancy images correspond to a silhouettes of the mean/motion compensated object shape and are used to obtain a visual hull reconstruction of the object. We show promising results on challenging monocular datasets of deforming objects where traditional visual hull intersection approaches fail to reconstruct the object correctly.

Despite the current availability in resource-rich regions of advanced technologies in scanning and 3-D imaging in current ophthalmology practice, world-wide screening tests for early detection and progression of glaucoma still consist of a variety of simple tools, including fundus image-based parameters such as CDR (cup to disc diameter ratio) and CAR (cup to disc area ratio), especially in resource -poor regions. Reliable automated computation of the relevant parameters from fundus image sequences requires robust non-rigid registration and segmentation techniques. Recent research work demonstrated that proper non-rigid registration of multi-view monocular fundus image sequences could result in acceptable segmentation of cup boundaries for automated computation of CAR and CDR. This research work introduces a composite diffeomorphic demons registration algorithm for segmentation of cup boundaries from a sequence of monocular images and compares the resulting CAR and CDR values with those computed manually by experts and from 3-D visualization of stereo pairs. Our preliminary results show that the automated computation of CDR and CAR from composite diffeomorphic segmentation of monocular image sequences yield values comparable with those from the other two techniques and thus may provide global healthcare with a cost-effective yet accurate tool for management of glaucoma in its early stage.

This paper introduces a new model-based approach for simultaneously reconstructing 3D human motion and full-body skeletal size from a small set of 2D image features tracked from uncalibrated monocular video sequences. The key idea of our approach is to construct a generative human motion model from a large set of preprocessed human motion examples to constrain the solution space of monocular human motion tracking. In addition, we learn a generative skeleton model from prerecorded human skeleton data to reduce ambiguity of the human skeleton reconstruction. We formulate the reconstruction process in a nonlinear optimization framework by continuously deforming the generative models to best match a small set of 2D image features tracked from a monocular video sequence. We evaluate the performance of our system by testing the algorithm on a variety of uncalibrated monocular video sequences.

This paper presents a novel and fast scheme to detect different body parts in human motion. Using monocular video sequences, trajectory estimation and body modeling of moving humans are combined in a co-operating processing architecture. More specifically, for every individual person, features of body ratio, silhouette and appearance are integrated into a hybrid model to detect body parts. The conventional assumption of upright body posture is not required. We also present a new algorithm for accurately finding the center point of the human body. The body configuration is finally described by a skeleton model. The feasibility and accuracy of the proposed scheme are analyzed by evaluating its performance for various sequences with different subjects and motion types (walking, pointing, kicking, leaping and falling). Our detection system achieves nearly real-time performance (around 10 frames/second).Keywordsmotion analysistrajectory estimationbody modelingobject detection

Human action recognition is used in areas such as surveillance, entertainment, and healthcare.This paper proposes a system to recognize both single and continuous human actions from monocular video sequences, based on 3D human modeling and cyclic hidden Markov models (CHMMs).First, for each frame in a monocular video sequence, the 3D coordinates of joints belonging to a human object, through actions of multiple cycles, are extracted using 3D human modeling techniques.The 3D coordinates are then converted into a set of geometrical relational features (GRFs) for dimensionality reduction and discrimination increase.For further dimensionality reduction, k-means clustering is applied to the GRFs to generate clustered feature vectors.These vectors are used to train CHMMs separately for different types of actions, based on the Baum-Welch re-estimation algorithm.For recognition of continuous actions that are concatenated from several distinct types of actions, a designed graphical model is used to systematically concatenate different separately trained CHMMs.The experimental results show the effective performance of our proposed system in both single and continuous action recognition problems.

Recritude: Reflections on Postural Ontology

We present an approach for joint inference of 3D scene structure and semantic labeling for monocular video. Starting with monocular image stream, our framework produces a 3D volumetric semantic + occupancy map, which is much more useful than a series of 2D semantic label images or a sparse point cloud produced by traditional semantic segmentation and Structure from Motion(SfM) pipelines respectively. We derive a Conditional Random Field (CRF) model defined in the 3D space, that jointly infers the semantic category and occupancy for each voxel. Such a joint inference in the 3D CRF paves the way for more informed priors and constraints, which is otherwise not possible if solved separately in their traditional frameworks. We make use of class specific semantic cues that constrain the 3D structure in areas, where multiview constraints are weak. Our model comprises of higher order factors, which helps when the depth is unobservable.We also make use of class specific semantic cues to reduce either the degree of such higher order factors, or to approximately model them with unaries if possible. We demonstrate improved 3D structure and temporally consistent semantic segmentation for difficult, large scale, forward moving monocular image sequences.

We propose a novel method for pose-invariant facial expression recognition from monocular video sequences that combines stochastic and deterministic search processes. We use the simple face model called variable-intensity template, which can be prepared with very little time and effort. We tackle the two issues found in previous work on the variable-intensity template: low accuracy in head pose estimation, and assumption violations due to external intensity changes such as illumination change. We mitigate these issues by introducing the deterministic approach into the stochastic approach implemented as a particle filter. Our experiment demonstrates significant improvements in recognition performance for horizontal and vertical head orientations in the range of ±40 degrees and ±20 degrees, respectively, from the frontal view.

In this paper we present a novel approach for mimicking expressions in 3D from a monocular video sequence. To this end, first we construct a high resolution semantic mesh model through automatic global and local registration of a low resolution range data. Such a model is represented using predefined set of control points in a compact fashion, and animated using radial basis functions. In order to recover the 2D positions of the 3D control points in the observed sequence, we use cascade Adaboost-driven search. The search space is reduced through the use of predictive expression modeling. The optimal configuration of the Adaboost responses is determined using combinatorial linear programming which enforces the anthropometric nature of the model. Then the displacement can be reproduced on any version of the model, registered on another face. Our method doesn't require dense stereo estimation and can then produce realistic animations, using any 3D model. Promising experimental results demonstrate the potential of our approach.

In Computer Vision, the method of using a single image to learn depth information through the CNNs has gradually improved, especially for the unsupervised learning methods which trains without the ground-truth are based on the image process and vision. However, the depth map generated from these methods are relative depth values instead of absolute depths, therefore these methods can't be well applied to practical applications.In this paper, we use the monocular video sequence as input, and get partial true depth value by using a prior conditions, so as to constrain the depth and the relative pose of the entire sequence with the "real" scale. Here we solve the previous scale ambiguous problem by fusing the geometry information, which correct the scale of the entire sequence by our rescaled module. This paper only uses monocular sequences to accomplish these tasks.Moreover, our method is specifically effective in the field of automated guided vehicles with only a camera as the requirement which reducing the dependence on other devices but improved the quality.

One of the persistent challenges in computer vision has been tracking objects under varying lighting conditions. In this paper we present a method for estimation of 3D motion of a rigid object from a monocular video sequence under arbitrary changes in the illumination conditions under which the video was captured. This is achieved by alternately estimating motion and illumination parameters using a generative model for integrating the effects of motion, illumination and structure within a unified mathematical framework. The motion is represented in terms of translation and rotation of the object centroid, and the illumination is represented using a spherical harmonics linear basis. The method does not assume any model for the variation of the illumination conditions - lighting can change slowly or drastically. For the multi-camera tracking scenario, we propose a new photometric constraint that is valid over the overlapping field of view between two cameras. This is similar in nature to the well-known epipolar constraint, except that it relates the photometric parameters, and can provide an additional constraint for illumination invariant multi-camera tracking. We demonstrate the effectiveness of our tracking algorithm on single and multi-camera video sequences under severe changes of lighting conditions.

In this paper, we propose a template-based 3D surface reconstruction system of non-rigid deformable objects from monocular video sequence. Firstly, we generate a semi-dense template of the target object with structure from motion method using a subsequence video. This video can be captured by rigid moving camera orienting the static target object or by a static camera observing the rigid moving target object. Then, with the reference template mesh as input and based on the framework of classical template-based methods, we solve an energy minimization problem to get the correspondence between the template and every frame to get the time-varying mesh to present the deformation of objects. The energy terms combine photometric cost, temporal and spatial smoothness cost as well as as-rigid-as-possible cost which can enable elastic deformation. In this paper, an easy and controllable solution to generate the semi-dense template for complex objects is presented. Besides, we use an effective iterative Schur based linear solver for the energy minimization problem. The experimental evaluation presents qualitative deformation objects reconstruction results with real sequences. Compare against the results with other templates as input, the reconstructions based on our template have more accurate and detailed results for certain regions. The experimental results show that the linear solver we used performs better efficiency compared to traditional conjugate gradient based solver.

The paper is focused on the problem of estimating 3D structure and motion of nonrigid object from a monocular video sequence. Many previous methods on this problem utilize the extension technique of factorization based on rank constraint to the tracking matrix, where the 3D shape of nonrigid object is expressed as weighted combination of a set of shape bases. All these solutions are based on the assumption of affine camera model. This assumption will become invalid and cause large reconstruction errors when the object is close to the camera. The main contribution of this paper is that we extend these methods to the general perspective camera model. The proposed algorithm iteratively updates the shape and motion from weak perspective projection to fully perspective projection by refining the scalars corresponding to the projective depths. Extensive experiments on real sequences validate the effectiveness and improvements of the proposed method.

In recent years, there has been a growing interest on tackling the Non-Rigid Structure from Motion problem (NRSfM), where the shape of a deformable object and the pose of a moving camera are simultaneously estimated from a monocular video sequence. Existing solutions are limited to single objects and continuous, smoothly changing sequences. In this paper we extend NRSfM to a multi-instance domain, in which the images do not need to have temporal consistency, allowing for instance, to jointly reconstruct the face of multiple persons from an unordered list of images. For this purpose, we present a new formulation of the problem based on a dual low-rank shape representation, that simultaneously captures the between- and within-individual deformations. The parameters of this model are learned using a variant of the probabilistic linear discriminant analysis that requires consecutive batches of expectation and maximization steps. The resulting approach estimates 3D deformable shape and pose of multiple instances from only 2D point observations on a collection images, without requiring pre-trained 3D data, and is shown to be robust to noisy measurements and missing points. We provide quantitative and qualitative evaluation on both synthetic and real data, and show consistent benefits compared to current state of the art.