3D facial model construction and animation from a single frontal face image

In this paper, we propose a highly automatic approach for 3D photorealistic face reconstruction from a single frontal image. The key point of our work is the implementation of adaptive manifold learning approach. Beforehand, an active appearance model (AAM) is trained for automatic feature extraction and adaptive locally linear embedding (ALLE) algorithm is utilized to reduce the dimensionality of the 3D database. Then, given an input frontal face image, the corresponding weights between 3D samples and the image are synthesized adaptively according to the AAM selected facial features. Finally, geometry reconstruction is achieved by linear weighted combination of adaptively selected samples. Radial basis function (RBF) is adopted to map facial texture from the frontal image to the reconstructed face geometry. The texture of invisible regions between the face and the ears is interpolated by sampling from the frontal image. This approach has several advantages: (1) Only a single frontal face image is needed for highly automatic face reconstruction; (2) Compared with former works, our reconstruction approach provides higher accuracy; (3) Constraint based RBF texture mapping provides natural appearance for reconstructed face.

Three-dimensional human face modeling is a very popular topic with many applications,such as facial animation, face recognition and model-based facial video communication. Therefore, how to model the facial geometry, texture intensity and illumination variation is important in computer vision and graphics. Previous works on 3D head modeling from a single face image utilized prior information on 3D head models. However, it is difficult to accurately reconstruct the 3D face model from a single face image with expression since the facial expression induces 3D face model deformation in a complex manner. The main challenge is the coupling of the neutral 3D face model and the 3D deformation due to expression, thus making the 3D model estimation from a single face image with expression very challenging. On the other hand, the illumination condition also makes the problem more difficult. In this thesis,we focus on developing a 3D face model reconstruction system including surface registration and training of 3D face models with expressional deformations as well as the estimation of the 3D neutral shape and the 3D expressional deformation from a single face image. The proposed reconstruction algorithm integrates the linear and non-linear subspace representations for a prior 3D neutral morphable model and the probabilistic manifold-based 3D expressional deformation. We incorporate the face geometry, expression deformation, texture and illumination information into the problem so that it is well constrained. The reconstructed 3D face models can also be further extended and applied to many real-world applications.

Image-based 3D shape reconstruction is essential for 3D modeling in recognition, virtual reality, generation of video games, 3D animation, and in the creation of 3D virtual fitting rooms - the sought-after application in this dissertation. In this work, we propose a novel reconstruction method that uses a generic 3D model that roughly controls the overall reconstruction shape which in turn, reduces the number of 2D images needed to just two (frontal and profile) for obtaining a 3D point model. The generic model consists of a massive set of 3D points that lack saliency, topological meaning, and relational interconnections. This is also true about points in the two canonical images. In this work, we propose finding a small set of interconnected ordered intrinsic salient points (control points) residing on the silhouette of the projections of the generic model onto the two image spaces, as well as finding those for the canonical images. Points on the generic model and the images are automatically given saliency with order and interconnections through loop-subdivision operating on those control points and their equivalent points on the canonical images. The transformation of the generic model into the specific object is done through morphing the points on the generic model to follow and be consistent with their equivalent points on the canonical images. The control points on the canonical images are automatically obtained using an active shape model (ASM) that modifies an average trained control point set attained using an offline training, to account for class shape variability and boundary regularization guaranteeing the control points to reside on the silhouette (boundary) of the individual. The reconstructed 3D model is furtherly refined through smoothing the reconstructed surface, while fixing local holes using a principle component analysis (PCA) decomposition. This reconstruction method is convenient, simple, efficient, and results in an average geometric error of less than 0.5% over 700 diverse body models from the CAESAR dataset. We then implement the contour-based articulated model to the personalized 3D model for pose recovery. The evolution of the model through the articulation process is captured in a video clip. 3D transformations are found and applied to the 3D model on its parts so that the projections of the 3D model 'match' those observed in the video sequence at corresponding frames. This is done by minimizing the error between the frontal projection body region points and the target points from the image for each independent moving part, with sub-resolution recovery errors. Once the personalized 3D model is obtained and the articulated model is implemented, the algorithm simulates how garments virtually appear on the reconstructed person under different poses. Different types of clothes are classified as tight or loose, where for the first type of garments, the garment is mapped onto the person's body, whereas for the second type, the mapping adjusts to accommodate the loose nature of the garment, and its fitting on the body of the reconstructed person model. This creates a virtual in-home fitting room for garment fitting, which is timely and is a needed application that would go hand in hand with online clothes shopping, allowing the trying of garments at home with changing poses before ordering them, and can be updated as often as necessary to accommodate changes in physique, weight, aging, etc. This is also done with minimum requirements from the user aside from taking two images and a video of pose articulations using their cam or smart phone, and stands as a useful, needed, and attractive App.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

A novel 3D facial animation model is proposed in this paper to generate realistic facial animation sequence. The 3D facial animation model is constructed based on morphable model and compatible with MPEG-4. Uniform mesh resampling method is put forward to align prototypic 3D faces and facial animation principle defined in MPEG-4 is adopted to drive the 3D facial animation model. The animation model can automatically reconstruct realistic 3D face of specific person given a single image and be driven by FAP parameters to generate 3D facial animation sequence automatically. Experimental results show the 3D facial animation model proposed in this paper can effectively generate high realistic facial animation sequence automatically.

We present an algorithm for 3D face modeling from a frontal image and a profile image of a person's face. The algorithm starts by computing the 3D coordinates of automatically extracted facial feature points. The coordinates of the selected feature points are then used to deform a 3D generic face model to obtain a 3D face model for that person. Procrustes analysis is used to minimize globally the distances between the facial feature vertices in the model and the corresponding 3D points obtained from the images. Then, local deformation is performed on the facial feature vertices to obtain a more realistic 3D model for the person. Preliminary experiments to asses the applicability of the models for face recognition show encouraging results.

In this paper, we present a system for synthesizing 3D human face models containing different expressions from a single facial image. Given a frontal image of the target face with neutral expression, we first detect several key points denoting the shape of the face by Active Shape Model (ASM). Then we apply a RBF-based scattered data interpolation to reconstruct a 3D target face using a neutral expression 3D face model as reference. By analyzing a series of 3D expression face model, we segment the 3D reference model into regions automatically that each region is correspondent to a facial organ. From the expression set we construct a motion model for each facial action with respect to the target face in a local consistent manner. At last, the reconstructed 3D target face model with neutral expression and the facial action motion model are combined to generate 3D target face of various expressions. There are 3 contributions of our work: (1) We employ a set of registered 3D facial expression models as input, which enabled us to generate more complex and visual-realistic expressions than other parameter-based approaches and 2D image-based methods. (2) On the basis of a clustering-based segmentation, we developed a localized linear expression model, which make it possible for us to generate different facial expressions both locally and globally, thusly enlarge the space of synthesize output and break the limitation by the limited scale of the input expression model set. (3) A local space transform procedure is included that the output expression can fit distinct facial shapes regardless of the scarcity of variation of the facial shapes (fat or thin) in the input model set.

This paper presents estimation of head pose angles from a single 2D face image using a 3D face model morphed from a reference face model. A reference model refers to a 3D face of a person of the same ethnicity and gender as the query subject. The proposed scheme minimizes the disparity between the two sets of prominent facial features on the query face image and the corresponding points on the 3D face model to estimate the head pose angles. The 3D face model used is morphed from a reference model to be more specific to the query face in terms of the depth error at the feature points. The morphing process produces a 3D face model more specific to the query image when multiple 2D face images of the query subject are available for training. The proposed morphing process is computationally efficient since the depth of a 3D face model is adjusted by a scalar depth parameter at feature points. Optimal depth parameters are found by minimizing the disparity between the 2D features of the query face image and the corresponding features on the morphed 3D model projected onto 2D space. The proposed head pose estimation technique was evaluated on two benchmarking databases: 1) the USF Human-ID database for depth estimation and 2) the Pointing'04 database for head pose estimation. Experiment results demonstrate that head pose estimation errors in nodding and shaking angles are as low as 7.93° and 4.65° on average for a single 2D input face image.

In this paper, a novel framework for 3D face reconstruction from a single 2D face image was proposed. We focus on generating 3D face model without expensive devices and complicated calculation. First, we preprocess 2D face image, including illumination compensation, face detection and feature point extraction. Then, the method is based on a 3D morphable face model that encodes shape and texture in terms of model parameters. The prior 3D face model is a linear combination of “eigenheads” obtained by applying PCA to a training set of laser-scanned 3D faces. To account for pose and illumination variations, the algorithm simulates the process of image formation in 3D face and it estimates 3D face and texture of faces from a single image. As opposed to previous literature, our method can locate facial feature points automatically in a 2D facial image. Moreover, we also show that our proposed method for pose estimation can be successfully applied to 3D face reconstruction. In our experiment results, the method proposed has a satisfied performance regarding calculating time and the reconstructing photorealistic 3D faces.

We propose a three-dimensional (3D) face-modelling method from a single two-dimensional (2D) face image using a gallery of 2D face images and their corresponding 3D face models. Unlike existing methods, which require human effort, we provide a simple way to reconstruct 3D face models without user interaction. Our main approach is based on the idea that a particular coefficient that linearly combines vectors of 2D face images and outputs a vector that approximates the input image vector in terms of the vector norm can be reused in 3D models. Therefore, the pair of a 2D image and its 3D model plays an important role in our algorithm. Using the FaceGen software allows us to avoid the employed in previous works procedure whereby the 3D model is generated in a labor-intensive, expensive way. As a result, we are able to easily establish our 2D and 3D database. In this paper, we present a method for adopting the coefficients in 3D models and demonstrate the results of our algorithm.

In this paper we describe a multi-modal ear and face biometric system. The system is comprised of two components: a 3D ear recognition component and a 2D face recognition component. For the 3D ear recognition, a series of frames is extracted from a video clip and the region of interest (i.e., ear) in each frame is independently reconstructed in 3D using Shape From Shading. The resulting 3D models are then registered using the iterative closest point algorithm. We iteratively consider each model in the series as a reference model and calculate the similarity between the reference model and every model in the series using a similarity cost function. Cross validation is performed to assess the relative fidelity of each 3D model. The model that demonstrates the greatest overall similarity is determined to be the most stable 3D model and is subsequently enrolled in the database. For the 2D face recognition, a set of facial landmarks is extracted from frontal facial images using the Active Shape Model. Then, the response of facial images to a series of Gabor filters at the locations of facial landmarks are calculated. The Gabor features (attributes) are stored in the database as the face model for recognition. The similarity between the Gabor features of a probe facial image and the reference models are utilized to determine the best match. The match scores of the ear recognition and face recognition modalities are fused to boost the overall recognition rate of the system. Experiments are conducted using a gallery set of 402 video clips and a probe of 60 video clips (images). As a result, a rank-one identification rate of 100% was achieved using the weighted sum technique for fusion.

We present an approach to identify non-cooperative individuals at a distance from a sequence of images using 3D face models. Most biometric features (such as fingerprints, hand shape, iris or retinal scans) require cooperative subjects in close proximity to the biometric system. Even top 2D face recognition systems today are only reliable at a distance when presented with frontal images. Our approach is accomplished by processing images acquired with an ultra-high resolution video camera, inferring the location of the subjects' head, using this information to crop the region of interest, building a 3D face model, and using this 3D model to perform biometric identification. To build the 3D model, we use an image sequence, as natural head and body motion provides enough viewpoint variation to perform stereo-motion for 3D face reconstruction. The 3D face model allows the use of true shape invariants for recognition against 3D galleries. Also the 3D model can assist in matching non-frontal faces against 2D galleries by rotating and rendering the 3D reconstruction into a 2D frontal image when only a 2D gallery is available. This approach constitutes a considerable step forward in solving the challenging problem of biometric identification at a distance.

When joint with a web cam, digitizer of image, & VLSI picture correlate chip, an IBM PC can be utilized for real-time facial identification. A semi-controlled lighting environment is used to capture a single frontal facial image, which is then preprocessed and template extracted before being correlated with a database of 173 photographs and finally post processed to yield identification. These steps use a single frontal facial image. Software development, VLSI hardware, picture preprocessing, and facial recognition are explored. The completely VLSI image corrugator can execute 340 Mop/second and increase 20 speed compared to adjusted assembly program execution on an 80486/66DX2. The algorithm recognizes a user from 173 photos of 34 persons in two to three seconds. Cross-authentication on the abstemiously variable databank yields a conservative 88% recognition rate for the system.

This paper proposes an efficient way of modeling 3D faces by using only two — a frontal and a profile — images. Although it is desirable to utilize only one single image for 3D face modeling, more accurate depth information can be obtained if we use a profile face image additionally. Despite this seemly straightforward task, however, no standard solutions for 3D face modeling with two images have yet been reported. To tackle this problem, in our work, we first extract facial shape information from each image and then align these two shapes in order to obtain a sparse 3D face. Then, the observed sparse 3D face is combined into generic dense depth information. By doing so, we reflect both the observed 3D sparse depth information and smooth depth changes around facial areas in our reconstructed 3D shape. Finally, the intensity of the frontal image is texture-mapped onto the reconstructed 3D shape for realistic 3D modeling. Unlike other 3D modeling methods, our proposed work is extremely fast (within a few seconds) and does not require any complex hardware settings or calibration. We illustrate our 3D modeling results by using the MPIE-database and demonstrate the effectiveness of the proposed approach.

Current appearance-based face recognition system encounters the difficulty to recognize faces with appearance variations, while only a small number of training images are available. We present a scheme based on the analysis by synthesis framework. A 3D generic face model is aligned onto a given frontal face image. A number of synthetic face images are generated with appearance variations from the aligned 3D face model. These synthesized images are used to construct an affine subspace for each subject. Training and test images for each subject are represented in the same way in such a subspace. Face recognition is achieved by minimizing the distance between the subspace of a test subject and that of each subject in the database. Only a single face image of each subject is available for training in our experiments. Preliminary experimental results are promising.