Disentangled Representation Learning for 3D Face Shape

Over the past few years, Generative Adversarial Networks (GANs) have garnered increased interest among researchers in Computer Vision, with applications including, but not limited to, image generation, translation, imputation, and super-resolution. Nevertheless, no GAN-based method has been proposed in the literature that can successfully represent, generate or translate 3D facial shapes (meshes). This can be primarily attributed to two facts, namely that (a) publicly available 3D face databases are scarce as well as limited in terms of sample size and variability (e.g., few subjects, little diversity in race and gender), and (b) mesh convolutions for deep networks present several challenges that are not entirely tackled in the literature, leading to operator approximations and model instability, often failing to preserve high-frequency components of the distribution. As a result, linear methods such as Principal Component Analysis (PCA) have been mainly utilized towards 3D shape analysis, despite being unable to capture non-linearities and high frequency details of the 3D face—such as eyelid and lip variations. In this work, we present 3DFaceGAN, the first GAN tailored towards modeling the distribution of 3D facial surfaces, while retaining the high frequency details of 3D face shapes. We conduct an extensive series of both qualitative and quantitative experiments, where the merits of 3DFaceGAN are clearly demonstrated against other, state-of-the-art methods in tasks such as 3D shape representation, generation, and translation.

Facial mask, an essential product for skincare, has become a part of daily life for female population. Existing facial mask design has a lot of misfits including missing regions, corrugation and misplacement. There is a need to build a relationship between a 3D face shape and a 2D mask template. An accurate 3D face model and anthropometric measurements are initial steps to design of masks. 3D surface flattening is a method to generate 2D patterns and is widely used to trim materials in garment and footwear industry. In this study, a manual generation of 2D flat pattern from a set of 3D Chinese female face models was described. Facial mask templates with different sizes and cuts were produced and evaluated. The results revealed that sizing dimensions mainly differed in the width for different face models. In addition, the location and area for eye and mouth regions had a relatively large deviation and therefore it should be paid more attention in the mask design. This study provided more insights into the relationship between 2D flat pattern and 3D face shape. The results were expected as a reference for facial mask design and improve its quality to fit.

This paper presents a novel method for face alignment under unknown head poses and nonrigid warp, within the framework of markov random field. The proposed method learns a 3D face shape model comprised of 31 facial features and a texture model for each facial feature from a 3D face database. The models are combined to serve as the unary, pairwise and high order constraints of the markov random field. The facial features are located by minimizing the potential function of markov random field, which is solved with dual decomposition. The main contribution of this paper is composed of three aspects. First, Random Project Tree is utilized to learn the manifold structure of the facial feature appearance under different view points. Second, a 3D face shape model is learned to capture the linear part of face shape distribution due to the change of identity and expression, which is served as the high order constraints. Third, Markov random field is introduced to model the nonlinear part of the face shape distribution, and also to deal with occlusion of facial features due to head pose variation or ornaments. Experiments was taken on the Texas 3D face database and face images downloaded from the Internet, results shows capability of adapting large head pose variations of the method.KeywordsFace ImageFacial FeatureMarkov Random FieldFace ShapeTexture ModelThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper describes a vector representation of 3D shapes acquired in the form of 2D range images produced using a range finder (NEC Danae-R). The vectorization was performed by first extracting feature points that correspond to each other as landmarks among different 3D face shapes, and then re-sampling each surface region of the face divided by the landmarks into other regions to obtain corresponding elements of the 3D shape vector. A linear description of the 3D shape spaces obtained by principal component analysis allows a smooth morphable modeling of a human face in terms of a smaller number of projection parameters. We scientifically solved the relations between shapes and impressions of faces and attempted face-image manipulations that transform impressions on facial attributes.

We propose a method that can reconstruct both a 3D facial shape and camera poses from freehand multi-viewpoint snapshots. This method is based on Active Shape Model (ASM) using a facial shape database. Most ASM methods require an image in which the camera pose is known, but our method does not require this information. First, we choose an initial shape by selecting the model from the database which is most suitable to input images. Then, we improve the model by morphing it to fit the input images. Next, we estimate the camera poses using the morphed model. Finally we repeat the process, improving both the facial shape and the camera poses until the error between the input images and the computed result is minimized. Through experimentation, we show that our method reconstructs the facial shape within 3.5 mm of the ground truth.KeywordsFeature PointInput ImageFacial ShapeActive Shape ModelWorld Coordinate SystemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We propose a method that can reconstruct both a 3D facial shape and camera poses from freehand multi-viewpoint snapshots. This method is based on Active Shape Model (ASM) using a facial shape database. Most ASM methods require an image in which the camera pose is known, but our method does not require this information. First, we choose an initial shape by selecting the model from the database which is most suitable to input images. Then, we improve the model by morphing it to fit the input images. Next, we estimate the camera poses using the morphed model. Finally we repeat the process, improving both the facial shape and the camera poses until the error between the input images and the computed result is minimized. Through experimentation, we show that our method reconstructs the facial shape within 3.5 mm of the ground truth.

Facial expression synthesizing is a process of generating new face shapes from a given face. The earlier work on synthesizing facial expressions used 2D images. Only recently, the work has moved to using 3D face shapes due to the availability and improvement of 3D scanner acquisition technology. This paper presents a work on synthesizing neutral facial expression or neutralizing facial expression on realistic 3D face models that holds distinct facial characteristics of the subjects. We apply Active Shape Model (ASM) to these dense face shapes. This approach could be used to improve the performance of existing biometric face identification systems, or to enhance human recognition.

Example‐based statistical face models using principle component analysis (PCA) have been widely deployed for three‐dimensional (3D) face reconstruction and face recognition. The two common factors that are generally concerned with such models are the size of the training dataset and the selection of different examples in the training set. The representational power (RP) of an example‐based model is its capability to depict a new 3D face for a given 2D face image. The RP of the model can be increased by correspondingly increasing the number of training samples. In this contribution, a novel approach is proposed to increase the RP of the 3D face reconstruction model by deforming a set of examples in the training dataset. A PCA‐based 3D face model is adapted for each new near frontal input face image to reconstruct the 3D face shape. Further an extended Tikhonov regularisation method has been employed to reconstruct 3D face shapes from a set of facial points. The results justify that the proposed adaptive PCA‐based model considerably improves the RP of the standard PCA‐based model and outperforms it with a 95% confidence level.

We address the problem of editing facial expression in video, such as exaggerating, attenuating or replacing the expression with a different one in some parts of the video. To achieve this we develop a tensor-based 3D face geometry reconstruction method, which fits a 3D model for each video frame, with the constraint that all models have the same identity and requiring temporal continuity of pose and expression. With the identity constraint, the differences between the underlying 3D shapes capture only changes in expression and pose. We show that various expression editing tasks in video can be achieved by combining face reordering with face warping, where the warp is induced by projecting differences in 3D face shapes into the image plane. Analogously, we show how the identity can be manipulated while fixing expression and pose. Experimental results show that our method can effectively edit expressions and identity in video in a temporally-coherent way with high fidelity.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. However, previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry, which is attributed to insufficient ground-truth 3D shapes, unreliable training strategies and limited representation power of 3DMM. To alleviate this issue, this paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person. Specifically, given a 2D image as the input, we virtually render the image in several calibrated views to normalize pose variations while preserving the original image geometry. A many-to-one hourglass network serves as the encode-decoder to fuse multiview features and generate vertex displacements as the fine-grained geometry. Besides, the neural network is trained by directly optimizing the visual effect, where two 3D shapes are compared by measuring the similarity between the multiview images rendered from the shapes. Finally, we propose to generate the ground-truth 3D shapes by registering RGB-D images followed by pose and shape augmentation, providing sufficient data for network training. Experiments on several challenging protocols demonstrate the superior reconstruction accuracy of our proposal on the face shape.

Cascaded regression has recently become the method of choice for solving non-linear least squares problems such as deformable image alignment. Given a sizeable training set, cascaded regression method learns a set of generic rules that are sequentially applied to solve the least squares problem. Despite the success of cascaded regression for problems such as face alignment and head pose estimation, there are several shortcomings arising in the strategies proposed thus far. For example, the result is sensitive to the initial shape and the storage of the model is too large, and no robustness to self-occlusion. This paper proposes an approach to solve that problem. First, we reconstruct the 3D face from a 2D face image with arbitrary poses and expressions. Second, we project the 3D face shape model to the 2D plane, then we can get a 2D face shape. Third, we use the projected 2D face shape as the initial shape for the cascaded regression instead of the generic mean shape. Comparing to the existing methods the proposed method can increase the accuracy of the face alignment under confirming appropriate performance.

Virtual orthognathic surgical planning involves simulating surgical corrections of jaw deformities on 3D facial bony shape models. Due to the lack of necessary guidance, the planning procedure is highly experience-dependent and the planning results are often suboptimal. A reference facial bony shape model representing normal anatomies can provide an objective guidance to improve planning accuracy. Therefore, we propose a self-supervised deep framework to automatically estimate reference facial bony shape models. Our framework is an end-to-end trainable network, consisting of a simulator and a corrector. In the training stage, the simulator maps jaw deformities of a patient bone to a normal bone to generate a simulated deformed bone. The corrector then restores the simulated deformed bone back to normal. In the inference stage, the trained corrector is applied to generate a patient-specific normal-looking reference bone from a real deformed bone. The proposed framework was evaluated using a clinical dataset and compared with a state-of-the-art method that is based on a supervised point-cloud network. Experimental results show that the estimated shape models given by our approach are clinically acceptable and significantly more accurate than that of the competing method.

Foetal alcohol syndrome (FAS) is a preventable condition caused by maternal alcohol consumption during pregnancy. The FAS facial phenotype is an important factor for diagnosis, alongside central nervous system impairments and growth abnormalities. Current methods for analysing the FAS facial phenotype rely on 3D facial image data, obtained from costly and complex surface scanning devices. An alternative is to use 2D images, which are easy to acquire with a digital camera or smart phone. However, 2D images lack the geometric accuracy required for accurate facial shape analysis. Our research offers a solution through the reconstruction of 3D human faces from single or multiple 2D images. We have developed a framework for evaluating 3D human face reconstruction from a single-input 2D image using a 3D face model for potential use in FAS assessment. We first built a generative morphable model of the face from a database of registered 3D face scans with diverse skin tones. Then we applied this model to reconstruct 3D face surfaces from single frontal images using a model-driven sampling algorithm. The accuracy of the predicted 3D face shapes was evaluated in terms of surface reconstruction error and the accuracy of FAS-relevant landmark locations and distances. Results show an average root mean square error of 2.62 mm. Our framework has the potential to estimate 3D landmark positions for parts of the face associated with the FAS facial phenotype. Future work aims to improve on the accuracy and adapt the approach for use in clinical settings. Significance: Our study presents a framework for constructing and evaluating a 3D face model from 2D face scans and evaluating the accuracy of 3D face shape predictions from single images. The results indicate low generalisation error and comparability to other studies. The reconstructions also provide insight into specific regions of the face relevant to FAS diagnosis. The proposed approach presents a potential cost-effective and easily accessible imaging tool for FAS screening, yet its clinical application needs further research.

It is not only interesting to predict how an individual of a relatively young age will look in the future but also to reconstruct the facial appearance in the past during childhood. It can be even more desirable when different circumstances, behavior and lifestyle and their impacts on the facial shape appearance as a consequence are taken into account. Such may be applicable for many practical reasons in healthcare, forensics psychology, missing people and children, etc. This paper presents the 3D Face Time Machine Matrix (FT2M), a 3D Dynamic Shape Model which is a fusion of two models of ageing and rejuvenation with facial shape variations due to lifestyle and behavioral factors. This dynamic model is learned from a database of three dimensional facial images which is built by ten individual age groups between 3 to 75 years old. 3D facial aging modeling is a complex process since it affects both the shape and texture of the face. We propose a Dynamic face model to transform the given input face to his youthful or adulthood appearance by taking into account his lifestyle and behavioral traits and the probable changes may occur in perceptible appearance by altering its shape and texture simultaneously.