Projective structure from facial motion

This study develops a 3D facial reconstruction system, which consists of five modules, using the orthographic projection model to approximate the perspective projection model. The first module identifies a number of feature points on the face and tracks these feature points over a sequence of facial images by the optical flow technique. The second module applies the factorization method to the orthographic model to reconstruct a 3D human face. The facial images are acquired using a pinhole camera, which are based on a perspective projection model. However, the face is reconstructed using an orthographic projection model. To compensate for the difference between these two models, the third module implements a simple and efficient method for approximating the perspective projection model. The fourth module overcomes the missing point problem, commonly arising in 3D reconstruction applications. Finally, the fifth module implements a smoothing process for the 3D surface by interpolating additional vertices.

Conclusions. Data about the recovery course of facial function after intratemporal facial nerve reconstruction using interposition nerve graft would provide useful information for clinicians to understand the regenerative process of the facial nerve after this type of surgery. It would also enable them to obtain informed consent from the patients by preoperatively explaining the predicted outcome of the postoperative facial paralysis. Objective. The purpose of this study was to describe the recovery course of facial movement and electrophysiological findings after intratemporal facial nerve reconstruction using interposition graft. Patients and methods. Five patients who underwent reconstruction of the facial nerve using interposition nerve graft immediately after facial nerve excision during surgery for temporal bone lesions were included in this study. Each patient was evaluated for facial movement (Yanagihara score), blink reflex (BR), and electroneurography (ENoG) preoperatively and postoperatively. Results. Improvement in facial movement began 8–10 months postoperatively. The score then gradually increased, and reached a plateau level by 2 years following surgery. The final score in four of the five patients ranged from 20 to 24 points, while the facial score of one patient only reached 12 points even at 3 years after surgery. All patients demonstrated moderate to severe synkinesis. The reappearance of R1 in BR occurred 7–10 months postoperatively, almost simultaneously with the beginning of recovery of facial movement. The latency of R1 on the operated side became shortened with increasing postoperative time, although it remained considerably longer than that on the unoperated side, even after 2 postoperative years. The onset of recovery of ENoG value (10–12 months postoperatively) was always delayed compared with the actual facial movement recovery and never returned to the level in the unoperated side.

With the introduction of the "metaverse" concept, digital human technology has garnered widespread attention, and how to quickly model a high-precision 3D human face has become a research focus in the field of digital humans. To address the issue of 3D Morphable Models (3DMM) being unable to reconstruct facial details, this paper proposes an animatable facial detail generation algorithm based on generative adversarial loss, which is used to restore more facial reliefs on the basic face obtained from three-dimensional facial reconstruction, achieving a highly realistic reconstruction of digital human faces. Firstly, this paper employs displacement mapping technology to recover facial detail information, constructing an autoencoder network to predict displacement maps from input images, controlling the vertices of the 3D model to express more relief details. Secondly, to effectively capture the mid-to-high frequency details of the face, generative adversarial loss is introduced to model the high-frequency attribute differences between images, jointly constrained with photometric losses and other content losses to ensure the accuracy of generated facial details. Lastly, to decouple static from dynamic facial details, this paper embeds 3DMM expression-related parameters into the generator model of displacement maps, and adopts an identity averaging strategy to constrain the encoder network to model only the static details of the face, whereas the dynamic details are provided by the 3DMM parameters, enabling the generation of dynamic facial details driven by expressions. The final experimental results show that the algorithm proposed in this paper demonstrates competitive performance in terms of model reconstruction quality and accuracy when compared to other state-of-the-art algorithms in facial detail generation. Additionally, the ability of the model to dynamically construct facial details was verified through expression transfer experiments, and the importance of generative adversarial loss was confirmed through ablation experiments.

This letter describes a novel hand–eye calibration technique based on reprojection error minimization. In contrast to traditional hand–eye calibration methods, the proposed method directly takes images of the calibration pattern and does not require to explicitly estimate the camera pose for each input image. The proposed method is implemented as a pose graph optimization problem, so that it can solve the estimation problem efficiently and robustly, and it can be easily extended for different projection models. It can deal with different camera models (e.g, X-ray cameras with a source-detector projection model) by changing the projection model. Through simulations, we validated that the proposed method shows a good estimation accuracy, and it can be applied to hand–eye calibration with a source-detector camera model. The experimental results with real robots show that the proposed method is applicable to real environments, and it improves the quality of a task that requires accurate hand–eye estimation, such as three-dimensional reconstruction.

This paper proposes a method to reconstruct 3D point clouds of a static scene in real-time by a moving depth sensor. Based on a base-line 3D reconstruction algorithm which fuses depth maps from the moving camera, a depth map enhancement module is embedded in the depth fusion stage to improve the level of details of reconstructed 3D models. Depth enhancements using the Sobel operator and the Laplacian operator are applied for speed and quality considerations. Experiments for real-time reconstruction of 3D scenes, especially the reconstruction of 3D human faces are provided to validate the proposed method. The reconstruction results are inspiring in their high definition compared with the original base-line algorithm.

Foetal alcohol syndrome (FAS) is a preventable condition caused by maternal alcohol consumption during pregnancy. The FAS facial phenotype is an important factor for diagnosis, alongside central nervous system impairments and growth abnormalities. Current methods for analysing the FAS facial phenotype rely on 3D facial image data, obtained from costly and complex surface scanning devices. An alternative is to use 2D images, which are easy to acquire with a digital camera or smart phone. However, 2D images lack the geometric accuracy required for accurate facial shape analysis. Our research offers a solution through the reconstruction of 3D human faces from single or multiple 2D images. We have developed a framework for evaluating 3D human face reconstruction from a single-input 2D image using a 3D face model for potential use in FAS assessment. We first built a generative morphable model of the face from a database of registered 3D face scans with diverse skin tones. Then we applied this model to reconstruct 3D face surfaces from single frontal images using a model-driven sampling algorithm. The accuracy of the predicted 3D face shapes was evaluated in terms of surface reconstruction error and the accuracy of FAS-relevant landmark locations and distances. Results show an average root mean square error of 2.62 mm. Our framework has the potential to estimate 3D landmark positions for parts of the face associated with the FAS facial phenotype. Future work aims to improve on the accuracy and adapt the approach for use in clinical settings. Significance: Our study presents a framework for constructing and evaluating a 3D face model from 2D face scans and evaluating the accuracy of 3D face shape predictions from single images. The results indicate low generalisation error and comparability to other studies. The reconstructions also provide insight into specific regions of the face relevant to FAS diagnosis. The proposed approach presents a potential cost-effective and easily accessible imaging tool for FAS screening, yet its clinical application needs further research.

General multi view reconstruction from affine or projective cameras has so far been solved most efficiently using methods of factorizing image data matrices into camera and scene parameters. This can be done directly for affine cameras[18] and after computing epipolar geometry for projective cameras [17]. A notorious problem has been the fact that these factorization methods require all points to be visible in all views. This paper presents alternative algorithms for general affine and projective views of multiple points where a) points and camera centers are computed as the nullspace of one linear system constructed from all the image data b) only three points have to be visible in all views. The latter requirement increases the flexibility and usefulness of 3D reconstruction from multiple views. In the case of projective views and unknown epipolar geometry, an additional algorithm is presented which initially assumes affine views and compensates iteratively for the perspective effects. In this paper affine cameras are represented in a projective framework which is novel and leads to a unified treatment of parallel and perspective projection in a single framework. The experiments cover a wide range of different camera motions and compare the presented algorithms to factorization methods, including approaches which handle missing data.KeywordsStructure from MotionLinear Multiple View ReconstructionMissing DataAffine and Projective Cameras

The vast majority of Shape-from-Polarization (SfP) methods work under the oversimplified assumption of using orthographic cameras. Indeed, it is still not well understood how to project the Stokes vectors when the incoming rays are not orthogonal to the image plane. We try to answer this question presenting a geometric model describing how a general projective camera captures the light polarization state. Based on the optical properties of a tilted polarizer, our model is implemented as a pre-processing operation acting on raw images, followed by a per-pixel rotation of the reconstructed normal field. In this way, all the existing SfP methods assuming orthographic cameras can behave like they were designed for projective ones. Moreover, our model is consistent with state-of-the-art forward and inverse renderers (like Mitsuba3 and ART), intrinsically enforces physical constraints among the captured channels, and handles demosaicing of DoFP sensors. Experiments on existing and new datasets demonstrate the accuracy of the model when applied to commercially available polarimetric cameras.

In this paper, the similar characteristics of human face has been used to relax the numbers of the input into one single face image, and reconstruct the 3D shape based on a couple statistical model. Moreover the lighting conditions of the single input image can be different from that of the training database. The experiment results have demonstrated the effectiveness of the proposed method.

Modern technologies have made the capture and sharing of digital video commonplace; the combination of modern smartphones, cloud storage, and social media platforms have enabled video to become a primary source of information for many people and institutions. As a result, it is important to be able to verify the authenticity and source of this information, including identifying the source camera model that captured it. While a variety of forensic techniques have been developed for digital images, less research has been conducted toward the forensic analysis of videos. In part, this is due to a lack of standard digital video databases, which are necessary to develop and evaluate state-of-the-art video forensic algorithms. In this paper, to address this need, we present the video authentication and camera identification (video-ACID) database, a large collection of videos specifically collected for the development of camera model identification algorithms. The video-ACID database contains over 12 000 videos from 46 physical devices representing 36 unique camera models. Videos in this database are hand collected in a diversity of real-world scenarios are unedited and have known and trusted provenance. In this paper, we describe the qualities, structure, and collection procedure of video-ACID, which includes clearly marked videos for evaluating camera model identification algorithms. Finally, we provide baseline camera model identification results on these evaluation videos using the state-of-the-art deep-learning techniques. The Video-ACID database is publicly available at misl.ece.drexel.edu/video-acid.

Although significant progress has been made in the field of speech-driven 3D facial animation recently, the speech-driven animation of an indispensable facial component, eye gaze, has been overlooked by recent research. This is primarily due to the weak correlation between speech and eye gaze, as well as the scarcity of audio-gaze data, making it very challenging to generate 3D eye gaze motion from speech alone. In this paper, we propose a novel data-driven method which can generate diverse 3D eye gaze motions in harmony with the speech. To achieve this, we firstly construct an audio-gaze dataset that contains about 14 hours of audio-mesh sequences featuring high-quality eye gaze motion, head motion and facial motion simultaneously. The motion data is acquired by performing lightweight eye gaze fitting and face reconstruction on videos from existing audio-visual datasets. We then tailor a novel speech-to-motion translation framework in which the head motions and eye gaze motions are jointly generated from speech but are modeled in two separate latent spaces. This design stems from the physiological knowledge that the rotation range of eyeballs is less than that of head. Through mapping the speech embedding into the two latent spaces, the difficulty in modeling the weak correlation between speech and non-verbal motion is thus attenuated. Finally, our TalkingEyes, integrated with a speech-driven 3D facial motion generator, can synthesize eye gaze motion, eye blinks, head motion and facial motion collectively from speech. Qualitative and quantitative evaluations, along with a perceptual user study, demonstrate the superiority of the proposed method in generating diverse and natural 3D eye gaze motions from speech. The project page of this paper is: https://lkjkjoiuiu.github.io/TalkingEyes_Home/.

In this paper, we propose a method for reconstructing 3D sequential patterns from multiple images without knowing exact image correspondences and without calibrating linear camera sensitivity parameters on intensity. The sequential pattern is defined as a series of colored 3D points. We assume that the series of the points are obtained in multiple images, but the correspondence of individual points is not known among multiple images. For reconstructing sequential patterns, we consider a camera projection model which combines geometric and photometric information of objects. Furthermore, we consider camera projections in the frequency space. By considering the multi-view relationship on the new projection model, we show that the 3D sequential patterns can be reconstructed without knowing exact correspondence of individual image points in the sequential patterns; moreover, the recovered 3D patterns do not suffer from changes in linear camera sensitivity parameters. The efficiency of the proposed method is tested using real images.

Efficient Scene Change Detection and Camera Motion Annotation for Video Classification

Photometric stereo is a fundamental approach in computer vision. At its core lies a set of image irradiance equations each taken with a different illumination. The vast majority of studies in this field have assumed orthography as the projection model. This paper re-examines the basic set of equations of photometric stereo, under an assumption of perspective projection. We show that the resulting system is linear (as is the case under the orthographic model; Nevertheless, the unknowns are different in the perspective case). We then suggest a simple reconstruction algorithm based on the perspective formulae, and compare it to its orthographic counterpart on synthetic as well as real images. This algorithm obtained lower error rates than the orthographic one in all of the error measures. These findings strengthen the hypothesis that a more realistic set of assumptions, the perspective one, improves reconstruction significantly.

Recovering 3D geometry from cameras in underwater applications involves the Refractive Structure-from-Motion problem where the non-linear distortion of light induced by a change of medium density invalidates the single viewpoint assumption. The pinhole-plus-distortion camera projection model suffers from a systematic geometric bias since refractive distortion depends on object distance. This leads to inaccurate camera pose and 3D shape estimation. To account for refraction, it is possible to use the axial camera model or to explicitly consider one or multiple parallel refractive interfaces whose orientations and positions with respect to the camera can be calibrated. Although it has been demonstrated that the refractive camera model is well-suited for underwater imaging, Refractive Structure-from-Motion remains particularly difficult to use in practice when considering the seldom studied case of a camera with a flat refractive interface. Our method applies to the case of underwater imaging systems whose entrance lens is in direct contact with the external medium. By adopting the refractive camera model, we provide a succinct derivation and expression for the refractive fundamental matrix and use this as the basis for a novel two-view reconstruction method for underwater imaging. For validation we use synthetic data to show the numerical properties of our method and we provide results on real data to demonstrate its practical application within laboratory settings and for medical applications in fluid-immersed endoscopy. We demonstrate our approach outperforms classic two-view Structure-from-Motion method relying on the pinhole-plus-distortion camera model.