Direct manipulation of blendshapes using a sketch-based interface

Three-dimensional human face modeling is a very popular topic with many applications,such as facial animation, face recognition and model-based facial video communication. Therefore, how to model the facial geometry, texture intensity and illumination variation is important in computer vision and graphics. Previous works on 3D head modeling from a single face image utilized prior information on 3D head models. However, it is difficult to accurately reconstruct the 3D face model from a single face image with expression since the facial expression induces 3D face model deformation in a complex manner. The main challenge is the coupling of the neutral 3D face model and the 3D deformation due to expression, thus making the 3D model estimation from a single face image with expression very challenging. On the other hand, the illumination condition also makes the problem more difficult. In this thesis,we focus on developing a 3D face model reconstruction system including surface registration and training of 3D face models with expressional deformations as well as the estimation of the 3D neutral shape and the 3D expressional deformation from a single face image. The proposed reconstruction algorithm integrates the linear and non-linear subspace representations for a prior 3D neutral morphable model and the probabilistic manifold-based 3D expressional deformation. We incorporate the face geometry, expression deformation, texture and illumination information into the problem so that it is well constrained. The reconstructed 3D face models can also be further extended and applied to many real-world applications.

We present an algorithm for 3D face modeling from a frontal image and a profile image of a person's face. The algorithm starts by computing the 3D coordinates of automatically extracted facial feature points. The coordinates of the selected feature points are then used to deform a 3D generic face model to obtain a 3D face model for that person. Procrustes analysis is used to minimize globally the distances between the facial feature vertices in the model and the corresponding 3D points obtained from the images. Then, local deformation is performed on the facial feature vertices to obtain a more realistic 3D model for the person. Preliminary experiments to asses the applicability of the models for face recognition show encouraging results.

Background: The scarcity of 3D facial models presents a significant hurdle for researchers and educators. Gathering such data demands substantial resources. Objective: To introduce an open-source 3D morphing software to generate 3D facial data sets for research and to provide a large sample data set that is based on synthetically generated 3D models. Methods: Software is developed to morph 3D facial models in bulk by altering landmark locations. Twenty synthetic 3D facial models are generated utilizing deep learning tools and 28 landmarks located on each. The measurements of synthetic models are confirmed to be realistic by comparing them with facial statistics. Several facial deformities and types are simulated at various magnitudes on 3D models to generate a large data set. Results: An open-source software and an open-access data set of 980 3D facial models, each with 28 landmark locations, are provided. Since the data set is based on synthetically generated 3D models, no institutional review board approval is required. Conclusion: The 3D morphing software and the large 3D data set are expected to benefit researchers and educators in the field of facial surgery and facial landmark detection.

We present an approach to identify non-cooperative individuals at a distance from a sequence of images using 3D face models. Most biometric features (such as fingerprints, hand shape, iris or retinal scans) require cooperative subjects in close proximity to the biometric system. Even top 2D face recognition systems today are only reliable at a distance when presented with frontal images. Our approach is accomplished by processing images acquired with an ultra-high resolution video camera, inferring the location of the subjects' head, using this information to crop the region of interest, building a 3D face model, and using this 3D model to perform biometric identification. To build the 3D model, we use an image sequence, as natural head and body motion provides enough viewpoint variation to perform stereo-motion for 3D face reconstruction. The 3D face model allows the use of true shape invariants for recognition against 3D galleries. Also the 3D model can assist in matching non-frontal faces against 2D galleries by rotating and rendering the 3D reconstruction into a 2D frontal image when only a 2D gallery is available. This approach constitutes a considerable step forward in solving the challenging problem of biometric identification at a distance.

This paper presents estimation of head pose angles from a single 2D face image using a 3D face model morphed from a reference face model. A reference model refers to a 3D face of a person of the same ethnicity and gender as the query subject. The proposed scheme minimizes the disparity between the two sets of prominent facial features on the query face image and the corresponding points on the 3D face model to estimate the head pose angles. The 3D face model used is morphed from a reference model to be more specific to the query face in terms of the depth error at the feature points. The morphing process produces a 3D face model more specific to the query image when multiple 2D face images of the query subject are available for training. The proposed morphing process is computationally efficient since the depth of a 3D face model is adjusted by a scalar depth parameter at feature points. Optimal depth parameters are found by minimizing the disparity between the 2D features of the query face image and the corresponding features on the morphed 3D model projected onto 2D space. The proposed head pose estimation technique was evaluated on two benchmarking databases: 1) the USF Human-ID database for depth estimation and 2) the Pointing'04 database for head pose estimation. Experiment results demonstrate that head pose estimation errors in nodding and shaking angles are as low as 7.93° and 4.65° on average for a single 2D input face image.

In this paper, a novel framework for 3D face reconstruction from a single 2D face image was proposed. We focus on generating 3D face model without expensive devices and complicated calculation. First, we preprocess 2D face image, including illumination compensation, face detection and feature point extraction. Then, the method is based on a 3D morphable face model that encodes shape and texture in terms of model parameters. The prior 3D face model is a linear combination of “eigenheads” obtained by applying PCA to a training set of laser-scanned 3D faces. To account for pose and illumination variations, the algorithm simulates the process of image formation in 3D face and it estimates 3D face and texture of faces from a single image. As opposed to previous literature, our method can locate facial feature points automatically in a 2D facial image. Moreover, we also show that our proposed method for pose estimation can be successfully applied to 3D face reconstruction. In our experiment results, the method proposed has a satisfied performance regarding calculating time and the reconstructing photorealistic 3D faces.

It is very important for virtual character designers to generate 3D face models interactively and creatively in real-time. This paper investigates the use of sketch-based interface in 3D face modeling and focuses on how to map 2D sketchy features onto a 3D model. A mapping mechanism on the basis of contour lines is proposed. Representative points of the input are extracted, which retain the global and local shape originalities. Then the outer and inner contours are extracted automatically from a given 3D face model (as a template model), and a set of feature points are determined according to the 2D representative points. The displacements between the 2D representatives and the 3D feature points are calculated and used as the control parameters of model deformation. A novel face model is finally created by means of deforming the given model. The experiments show the efficiency and effectiveness of our approaches.KeywordsFeature PointFace ModelOuter ContourVirtual CharacterFace MeshThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A novel 3D facial animation model is proposed in this paper to generate realistic facial animation sequence. The 3D facial animation model is constructed based on morphable model and compatible with MPEG-4. Uniform mesh resampling method is put forward to align prototypic 3D faces and facial animation principle defined in MPEG-4 is adopted to drive the 3D facial animation model. The animation model can automatically reconstruct realistic 3D face of specific person given a single image and be driven by FAP parameters to generate 3D facial animation sequence automatically. Experimental results show the 3D facial animation model proposed in this paper can effectively generate high realistic facial animation sequence automatically.

The purpose of this study is to present a virtual planning protocol based on the database and three-dimensional (3D) image registration technology for the restoration of the total nasal defect, and evaluate its feasibility and clinical efficacy. Patients were scanned with a FaceScan to obtain the 3D facial model which was stored in an Extensible Neuroimaging Archive Toolkit (XNAT) database. Personalized search and similarity evaluation were performed in the database to find a normal 3D facial model with the highest similarity to the patient's. Then, the 2 models were imported into the dedicated software for 3D image registration to get the 3D model of the nasal defect part and the preoperative planning 3D facial model of virtual restoration. Subsequently, the dimensionality reduction algorithm was conducted to transform the 3D model of the nasal defect to a 2D flatten one for determining the scope of the forehead flap during surgery. Four weeks after the insetting surgery of forehead flap pedicle, the postoperative 3D facial model was gained. At last, the clinical efficacy was evaluated by comparing the difference between the preoperative planning and postoperative 3D facial model. The nasal shapes of the patients were good after the operation, and the results revealed that the maximum error was ranging from 3.12 to 4.07 mm with the mean error from 0.92 to 1.04 mm. The database and 3D image registration technology provide a new approach for precisely determining the scope of total nasal defect and the forehead flap, which may be used for reference in the accurate restoration of other facial soft tissue defects.

The use of video sequences for face recognition has been relatively less studied than image-based approaches. In this paper, we present a framework for face recognition from video sequences that is robust to large changes in facial pose and lighting conditions. Our method is based on a recently obtained theoretical result that can integrate the effects of motion, lighting and shape in generating an image using a perspective camera. This result can be used to estimate the pose and illumination conditions for each frame of the probe sequence. Then, using a 3D face model, we synthesize images corresponding to the pose and illumination conditions estimated in the probe sequences. Similarity between the synthesized images and the probe video is computed by integrating over the entire sequence. The method can handle situations where the pose and lighting conditions in the training and testing data are completely disjoint.

We introduce a facial expression training system using the bilinear shape model which helps people to practice making a facial expression. The user face on the camera preview screen is reconstructed into a 3D face model and the model is transformed to blend shape model which represents the facial expression. This way, the system can precisely analyze the facial expression of the user. With target 3D face model appearing on the screen, the 3D face model changes its facial expression, it leads the user to change his facial expression to become look like same. The system recognizes whether the facial expression of the user is same with the one of 3D face model. As the system gives the various missions to user to change his facial expression, user can practice the facial expression. It can be used for bell's palsy patient who needs face rehabilitation exercise or someone who need to practice unique facial expression such as stewardess smile or facial mimicry.

최근 컴퓨터 비전, 컴퓨터 그래픽스 분야의 급속한 발전과 하드웨어의 우수한 성능으로 인해 3차원 모델링 기술 또한 급속히 발전 되었다. 그리고 다양한 3D 컨텐츠들의 등장으로 3차원 모델링 기술의 수요가 더욱 많아지고 그 질 또한 높아지고 있다. 3D 얼굴 모델은 이러한 컨텐츠들에 폭 넓게 적용될 수 있어 활용성이 높다. 본 논문에서는 주어진 한 장의 2D 정면 얼굴 영상으로부터 3D 얼굴을 모델링하는 방법을 제안한다. 이를 위해 다수의 정점으로 구성된 3D 일반 얼굴 모델을 입력 얼굴 영상과의 대응관계를 통해 변형하여 3D 얼굴을 모델링한다. 우선 AAM을 이용하여 입력된 얼굴 영상에서 특징점을 추출한다. 그리고 추출된 특징점만으로 직관적인 모델을 변형하기 위해 메쉬 워핑을 적용한다. 여기에 얼굴의 깊이 정보를 유도하기 위하여 한 장의 정면 얼굴 영상을 활용한 명도 변화 기반의 깊이 값의 추출을 시도한다. 이러한 과정을 통해 경제적으로 부담되지 않으며 사용 환경에 대한 제한이 적어 보편적으로 활용될 수 있는 3D 얼굴 모델링 방법을 구현하고 그 타당성을 실험으로 보인다. Recently the 3D modeling techniques were developed rapidly due to rapid development of computer vision, computer graphics with the excellent performance of hardware. With the advent of a variety of 3D contents, 3D modeling technology becomes more in demand and it's quality is increased. 3D face models can be applied widely to such contents with high usability. In this paper, a 3D face modeling is attempted from a given single 2D frontal face image. To achieve the goal, we thereafter the feature points using AAM are extracted from the input frontal face image. With the extracted feature points we deform the 3D general model by 2-pass mesh warping, and also the depth extraction based on intensity values is attempted to. Throughout those processes, a universal 3D face modeling method with less expense and less restrictions to application environment was implemented and it's validity was shown through experiments.

We present an approach to identify non-cooperative individuals at a distance from a sequence of images using 3D face models. Most biometric features (such as fingerprints, hand shape, iris or retinal scans) require cooperative subjects in close proximity to the biometric system. We process images acquired with an ultra-high resolution video camera, infer the location of the subjects' head, use this information to crop the region of interest, build a 3D face model, and use this 3D model to perform biometric identification. To build the 3D model, we use an image sequence, as natural head and body motion provides enough viewpoint variation to perform stereo-motion for 3D face reconstruction. Experiments using a 3D matching engine suggest the feasibility of proposed approach for recognition against 3D galleries.

The vision of the surrounding and people that are within eyeshot influences the human well-being and safety. The rationale of system development that allows recognizing faces from difficult perspectives online and informing timely about approaching people is undisputed. The manuscript describes the methods of automatic detection of equilibrium face points in the bitmap image and methods of forming 3D face model. The optimal search algorithm for equilibrium points has been chosen. The method of forming 3D face model basing on a single bitmap image and building up the face image rotated to the preset angle has been proposed. The algorithm for estimating the angle and algorithm of the face image rotation have been implemented. The manuscript also reviews the existing methods of forming 3D face model. The algorithm for the formation of 3D face model from a single bitmap image and a set of individual 3D models have been proposed as well as the algorithm for forming different face angles with the calculated 3D face model aimed to create biometric vectors cluster. Operation results of the algorithm for face images formation from different angles have been presented.

In this paper, a low cost least squares optimization method for sub-Nyquist rate output of a two-phase holding (TPH) DAC is proposed. The pseudoinverse matrix of the TPH mode DAC reconstruction matrix is analyzed, and based on the analysis, a direct data manipulation method is proposed to replace matrix operations for reducing the computational complexity. The proposed least squares optimization method is able to improve the spurious free dynamic range by suppressing unwanted signals inside the bandwidth of interest. Simulation results show that the computational complexity can be reduced by six orders of magnitude.