Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape.

3D morphable model (3DMM) fitting on 2D data is traditionally done via unconstrained optimization with regularization terms to ensure that the result is a plausible face shape and is consistent with a set of 2D landmarks. This paper presents inequality-constrained 3DMM fitting as the first alternative to regularization in optimization-based 3DMM fitting. Inequality constraints on the 3DMM's shape coefficients ensure face-like shapes without modifying the objective function for smoothness, thus allowing for more flexibility to capture person-specific shape details. Moreover, inequality constraints on landmarks increase robustness in a way that does not require per-image tuning. We show that the proposed method stands out with its ability to estimate person-specific face shapes by jointly fitting a 3DMM to multiple frames of a person. Further, when used with a robust objective function, namely gradient correlation, the method can work "in-the-wild" even with a 3DMM constructed from controlled data. Lastly, we show how to use the log-barrier method to efficiently implement the method. To our knowledge, we present the first 3DMM fitting framework that requires no learning yet is accurate, robust, and efficient. The absence of learning enables a generic solution that allows flexibility in the input image size, interchangeable morphable models, and incorporation of camera matrix.

Foetal alcohol syndrome (FAS) is a preventable condition caused by maternal alcohol consumption during pregnancy. The FAS facial phenotype is an important factor for diagnosis, alongside central nervous system impairments and growth abnormalities. Current methods for analysing the FAS facial phenotype rely on 3D facial image data, obtained from costly and complex surface scanning devices. An alternative is to use 2D images, which are easy to acquire with a digital camera or smart phone. However, 2D images lack the geometric accuracy required for accurate facial shape analysis. Our research offers a solution through the reconstruction of 3D human faces from single or multiple 2D images. We have developed a framework for evaluating 3D human face reconstruction from a single-input 2D image using a 3D face model for potential use in FAS assessment. We first built a generative morphable model of the face from a database of registered 3D face scans with diverse skin tones. Then we applied this model to reconstruct 3D face surfaces from single frontal images using a model-driven sampling algorithm. The accuracy of the predicted 3D face shapes was evaluated in terms of surface reconstruction error and the accuracy of FAS-relevant landmark locations and distances. Results show an average root mean square error of 2.62 mm. Our framework has the potential to estimate 3D landmark positions for parts of the face associated with the FAS facial phenotype. Future work aims to improve on the accuracy and adapt the approach for use in clinical settings. Significance: Our study presents a framework for constructing and evaluating a 3D face model from 2D face scans and evaluating the accuracy of 3D face shape predictions from single images. The results indicate low generalisation error and comparability to other studies. The reconstructions also provide insight into specific regions of the face relevant to FAS diagnosis. The proposed approach presents a potential cost-effective and easily accessible imaging tool for FAS screening, yet its clinical application needs further research.

Facial expression synthesizing is a process of generating new face shapes from a given face. The earlier work on synthesizing facial expressions used 2D images. Only recently, the work has moved to using 3D face shapes due to the availability and improvement of 3D scanner acquisition technology. This paper presents a work on synthesizing neutral facial expression or neutralizing facial expression on realistic 3D face models that holds distinct facial characteristics of the subjects. We apply Active Shape Model (ASM) to these dense face shapes. This approach could be used to improve the performance of existing biometric face identification systems, or to enhance human recognition.

In this paper, a novel framework for 3D face reconstruction from a single 2D face image was proposed. We focus on generating 3D face model without expensive devices and complicated calculation. First, we preprocess 2D face image, including illumination compensation, face detection and feature point extraction. Then, the method is based on a 3D morphable face model that encodes shape and texture in terms of model parameters. The prior 3D face model is a linear combination of “eigenheads” obtained by applying PCA to a training set of laser-scanned 3D faces. To account for pose and illumination variations, the algorithm simulates the process of image formation in 3D face and it estimates 3D face and texture of faces from a single image. As opposed to previous literature, our method can locate facial feature points automatically in a 2D facial image. Moreover, we also show that our proposed method for pose estimation can be successfully applied to 3D face reconstruction. In our experiment results, the method proposed has a satisfied performance regarding calculating time and the reconstructing photorealistic 3D faces.

Example‐based statistical face models using principle component analysis (PCA) have been widely deployed for three‐dimensional (3D) face reconstruction and face recognition. The two common factors that are generally concerned with such models are the size of the training dataset and the selection of different examples in the training set. The representational power (RP) of an example‐based model is its capability to depict a new 3D face for a given 2D face image. The RP of the model can be increased by correspondingly increasing the number of training samples. In this contribution, a novel approach is proposed to increase the RP of the 3D face reconstruction model by deforming a set of examples in the training dataset. A PCA‐based 3D face model is adapted for each new near frontal input face image to reconstruct the 3D face shape. Further an extended Tikhonov regularisation method has been employed to reconstruct 3D face shapes from a set of facial points. The results justify that the proposed adaptive PCA‐based model considerably improves the RP of the standard PCA‐based model and outperforms it with a 95% confidence level.

This paper presents a novel method for face alignment under unknown head poses and nonrigid warp, within the framework of markov random field. The proposed method learns a 3D face shape model comprised of 31 facial features and a texture model for each facial feature from a 3D face database. The models are combined to serve as the unary, pairwise and high order constraints of the markov random field. The facial features are located by minimizing the potential function of markov random field, which is solved with dual decomposition. The main contribution of this paper is composed of three aspects. First, Random Project Tree is utilized to learn the manifold structure of the facial feature appearance under different view points. Second, a 3D face shape model is learned to capture the linear part of face shape distribution due to the change of identity and expression, which is served as the high order constraints. Third, Markov random field is introduced to model the nonlinear part of the face shape distribution, and also to deal with occlusion of facial features due to head pose variation or ornaments. Experiments was taken on the Texas 3D face database and face images downloaded from the Internet, results shows capability of adapting large head pose variations of the method.KeywordsFace ImageFacial FeatureMarkov Random FieldFace ShapeTexture ModelThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we present a novel strategy to design disentangled 3D face shape representation. Specifically, a given 3D face shape is decomposed into identity part and expression part, which are both encoded and decoded in a nonlinear way. To solve this problem, we propose an attribute decomposition framework for 3D face mesh. To better represent face shapes which are usually nonlinear deformed between each other, the face shapes are represented by a vertex based deformation representation rather than Euclidean coordinates. The experimental results demonstrate that our method has better performance than existing methods on decomposing the identity and expression parts. Moreover, more natural expression transfer results can be achieved with our method than existing methods.

T3D face reconstruction from a single 2D image is mathematically ill-posed. However, to solve ill-posed problems in the area of computer vision, a variety of methods has been proposed; some of the solutions are to estimate latent information or to apply model based approaches. In this paper, we propose a novel method to reconstruct a 3D face from a single 2D face image based on pose estimation and a deformable model of 3D face shape. For 3D face reconstruction from a single 2D face image, it is the first task to estimate the depth lost by 2D projection of 3D faces. Applying the EM algorithm to facial landmarks in a 2D image, we propose a pose estimation algorithm to infer the pose parameters of rotation, scaling, and translation. After estimating the pose, much denser points are interpolated between the landmark points by a 3D deformable model and barycentric coordinates. As opposed to previous literature, our method can locate facial feature points automatically in a 2D facial image. Moreover, we also show that the proposed method for pose estimation can be successfully applied to 3D face reconstruction. Experiments demonstrate that our approach can produce reliable results for reconstructing photorealistic 3D faces.

3D Morphable Models (3DMMs) are statistical models that represent facial texture and shape variations using a set of linear bases and more particular Principal Component Analysis (PCA). 3DMMs were used as statistical priors for reconstructing 3D faces from images by solving non-linear least square optimization problems. Recently, 3DMMs were used as generative models for training non-linear mappings (\ie, regressors) from image to the parameters of the models via Deep Convolutional Neural Networks (DCNNs). Nevertheless, all of the above methods use either fully connected layers or 2D convolutions on parametric unwrapped UV spaces leading to large networks with many parameters. In this paper, we present the first, to the best of our knowledge, non-linear 3DMMs by learning joint texture and shape auto-encoders using direct mesh convolutions. We demonstrate how these auto-encoders can be used to train very light-weight models that perform Coloured Mesh Decoding (CMD) in-the-wild at a speed of over 2500 FPS.

It is not only interesting to predict how an individual of a relatively young age will look in the future but also to reconstruct the facial appearance in the past during childhood. It can be even more desirable when different circumstances, behavior and lifestyle and their impacts on the facial shape appearance as a consequence are taken into account. Such may be applicable for many practical reasons in healthcare, forensics psychology, missing people and children, etc. This paper presents the 3D Face Time Machine Matrix (FT2M), a 3D Dynamic Shape Model which is a fusion of two models of ageing and rejuvenation with facial shape variations due to lifestyle and behavioral factors. This dynamic model is learned from a database of three dimensional facial images which is built by ten individual age groups between 3 to 75 years old. 3D facial aging modeling is a complex process since it affects both the shape and texture of the face. We propose a Dynamic face model to transform the given input face to his youthful or adulthood appearance by taking into account his lifestyle and behavioral traits and the probable changes may occur in perceptible appearance by altering its shape and texture simultaneously.

In this thesis we present two applications where 3D surface modeling and registration play a role of paramount importance. The first application is in archaeology, where our proposed method assists the archaeologist in the tedious process of reconstructing ceramic vessels from the excavated fragments (or shards). The method takes advantage of the surface intrinsic features coupled with a library of generic models constructed by the archaeologists to produce a virtual reconstruction of what the original vessel may have looked like. Generic models are generated based on the expert historical knowledge of the period, provenance of the artifact, and location of the digging site. The generic models need not to be identical to the original vessel, but must be within a geometric transformation of it in most of its parts. The methods are suited for ceramic vessels with distinctive color patterns or reliefs (i.e., surface with molding, carving, or stamping). We introduce a novel curvature B-spline fitting method that makes the extraction of robust features possible. The alignment of the shards against the generic model uses a novel set of 3D weighted curve moments and absolute affine invariants derived from them. The transformation is computed from corresponding parabolic contours on the shard and the generic model. When the shards have no color or relief on them, they are virtually mended to abutting shards using intrinsic differential anchor points computed on the shards break curves and a set of absolute invariants. For axially symmetric vessels, a global constraint induced by the surface of revolution is applied to guarantee global mending consistency. In our experiment involving around 300 shards, more than 85% of them are successfully mended. The mended shards consist of over 90% surface areas of the ceramic vessels. The second application is in 3D human face synthesis and recognition, where our proposed method assists the national security agencies in constructing a 3D face database to be used for spotting missing personnel or identify possible suspects when close up images of that person are flagged up. A gender/race generic 3D face model is morphed into a personalized unique 3D face of a given person from using a limited number of 2D images of the given person taken from different viewpoints. The 2D images need not to be of the same resolution, or taken under similar brightness or same place, or even same time period. The method applies also to drawing sketches of the person in question. The reconstructed 3D personalized model is an accurate rendition of the actual human face, and the reconstruction errors are smaller than a 3D scanner resolution that is used to scan the person's face. Our method renders a difficult 2D face recognition problem into a much simpler 3D surface registration (alignment) problem. Given a set of images of a test person, a personalized 3D face model is constructed and "best" aligned with one of the 3D faces stored in the database. When the minimum alignment errors are below an accepted threshold (lower than the scanner resolution), the identity of the test person is matched with the one in the database. Experimental results are shown in controlled environment with high resolution images, under ideal illumination condition, as well as for images taken in uncontrolled environment with low resolution, arbitrary illumination. In both conditions, the recognition rate is over 95%. It is also proved that better face recognition rate is achieved by using a gender/race based generic 3D face model which allows for accurate pose estimation as well as the face identification in 3D.

Facial mask, an essential product for skincare, has become a part of daily life for female population. Existing facial mask design has a lot of misfits including missing regions, corrugation and misplacement. There is a need to build a relationship between a 3D face shape and a 2D mask template. An accurate 3D face model and anthropometric measurements are initial steps to design of masks. 3D surface flattening is a method to generate 2D patterns and is widely used to trim materials in garment and footwear industry. In this study, a manual generation of 2D flat pattern from a set of 3D Chinese female face models was described. Facial mask templates with different sizes and cuts were produced and evaluated. The results revealed that sizing dimensions mainly differed in the width for different face models. In addition, the location and area for eye and mouth regions had a relatively large deviation and therefore it should be paid more attention in the mask design. This study provided more insights into the relationship between 2D flat pattern and 3D face shape. The results were expected as a reference for facial mask design and improve its quality to fit.

Recovering 3D shapes of deformable objects from single 2D images is an extremely challenging and ill-posed problem. Most existing approaches are based on structure-from-motion or graph inference, where a 3D shape is solved by fitting 2D keypoints/mask instead of directly using the vital cue in the original 2D image. These methods usually require multiple views of an object instance and rely on accurate labeling, detection, and matching of 2D keypoints/mask across multiple images. To overcome these limitations, we make effort to reconstruct 3D deformable object shapes directly from the given unconstrained 2D images. In training, instead of using multiple images per object instance, our approach relaxes the constraint to use images from the same object category (with one 2D image per object instance). The key is to disentangle the category-specific representation of the 3D shape identity and the instance-specific representation of the 3D shape displacement from the 2D training images. In testing, the 3D shape of an object can be reconstructed from the given image by deforming the 3D shape identity according to the 3D shape displacement. To achieve this goal, we propose a novel convolutional encoder-decoder network—the Disentangling Deep Network (DisDN). To demonstrate the effectiveness of the proposed approach, we implement comprehensive experiments on the challenging PASCAL VOC benchmark and use different 3D shape ground-truth in training and testing to avoiding overfitting. The obtained experimental results show that DisDN outperforms other state-of-the-art and baseline methods.

Over the past few years, Generative Adversarial Networks (GANs) have garnered increased interest among researchers in Computer Vision, with applications including, but not limited to, image generation, translation, imputation, and super-resolution. Nevertheless, no GAN-based method has been proposed in the literature that can successfully represent, generate or translate 3D facial shapes (meshes). This can be primarily attributed to two facts, namely that (a) publicly available 3D face databases are scarce as well as limited in terms of sample size and variability (e.g., few subjects, little diversity in race and gender), and (b) mesh convolutions for deep networks present several challenges that are not entirely tackled in the literature, leading to operator approximations and model instability, often failing to preserve high-frequency components of the distribution. As a result, linear methods such as Principal Component Analysis (PCA) have been mainly utilized towards 3D shape analysis, despite being unable to capture non-linearities and high frequency details of the 3D face—such as eyelid and lip variations. In this work, we present 3DFaceGAN, the first GAN tailored towards modeling the distribution of 3D facial surfaces, while retaining the high frequency details of 3D face shapes. We conduct an extensive series of both qualitative and quantitative experiments, where the merits of 3DFaceGAN are clearly demonstrated against other, state-of-the-art methods in tasks such as 3D shape representation, generation, and translation.

The use of three-dimensional (3D) facial scans for facial analysis is increasing in maxillofacial treatment. The aim of this study was to investigate the consistency of two-dimensional (2D) and 3D facial analyses performed by multiple raters. Six men and four women (25–36-year-old) participated in this study. The 2D images of the smiling and resting faces in the frontal and sagittal planes were obtained. The 3D facial and intraoral scans were merged to generate virtual 3D faces. Ten clinicians performed facial analyses by investigating 14 indices of 2D and 3D faces. Intra- and interrater agreements of the results of 2D and 3D facial analyses within and among the participants were evaluated. The intrarater agreement between the 2D and 3D facial analyses varied according to the indices. The highest and lowest agreements were found for the dental crowding index (0.94) and smile line curvature index (0.56) in the frontal plane, and Angle's classification (canine) index (0.98) and occlusal plane angle index (0.55) in the profile plane. In the frontal plane, the interrater agreements were generally higher for the 3D images than for the 2D images, while in the profile plane, the interrater agreements were high in the Angle's classification (canine) index however low in the other indices. Several occlusion-related indices were missing in the 2D images because the posterior teeth were not observed. Esthetic analysis results between 2D and 3D face images can differ according to the evaluation indices. The use of 3D faces is recommended over 2D images to increase the reliability of facial analyses, as it can fully assess both esthetic and occlusion-related indices.