Head Pose Estimation with Improved Random Regression Forests

We developed a fast and robust head pose and gaze estimation system. This system can detect facial feature points and estimate 3D pose angles and gaze direction in various conditions including changes in facial expression, partial occlusion, etc. The system needs only one face image as input and doesn't need any special devices such as blinking LED or stereo camera. Moreover, no calibration process is needed. It shows 95% of head pose estimation accuracy and 81% of gaze estimation accuracy (when the error margin is 15 degrees). The processing time is approximately 15ms/frame (Pentium4 3.2 GHz). Acceptable range of facial pose is within +/- 60 degrees in yaw (left-right) and within +/- 30 degrees in pitch (up-down).

This paper presents estimation of head pose angles from a single 2D face image using a 3D face model morphed from a reference face model. A reference model refers to a 3D face of a person of the same ethnicity and gender as the query subject. The proposed scheme minimizes the disparity between the two sets of prominent facial features on the query face image and the corresponding points on the 3D face model to estimate the head pose angles. The 3D face model used is morphed from a reference model to be more specific to the query face in terms of the depth error at the feature points. The morphing process produces a 3D face model more specific to the query image when multiple 2D face images of the query subject are available for training. The proposed morphing process is computationally efficient since the depth of a 3D face model is adjusted by a scalar depth parameter at feature points. Optimal depth parameters are found by minimizing the disparity between the 2D features of the query face image and the corresponding features on the morphed 3D model projected onto 2D space. The proposed head pose estimation technique was evaluated on two benchmarking databases: 1) the USF Human-ID database for depth estimation and 2) the Pointing'04 database for head pose estimation. Experiment results demonstrate that head pose estimation errors in nodding and shaking angles are as low as 7.93° and 4.65° on average for a single 2D input face image.

International audience

This paper presents a face quality assessment approach that selects the highest‐quality face image using a two‐stage process from video streaming. In high‐traffic environments, traditional face recognition methods can cause crowd congestion, emphasizing the need for unconscious face recognition, which requires no active cooperation from individuals. Due to the nature of unconscious face recognition, it is necessary to capture high‐quality face images. In this paper, the FSA‐Net head pose estimation network is enhanced to FSA‐Shared_Nadam by replacing the Adam optimizer with Nadam and improving stage fusion. In the first stage, FSA‐Shared_Nadam estimates head pose angles, MediaPipe detects facial landmarks to calculate eye distance and aspect ratios, and sharpness is calculated using the Laplacian operator. Images are considered valid if they meet the criteria. A model trains a face quality scoring formula, learning how different head pose angles affect face recognition accuracy. In the second stage, face images are clustered, and the formula is applied to select the highest‐scoring face within each cluster. The approach was tested across multiple datasets, and a simulated security checkpoint scenario was created for practical testing. The results demonstrate the effectiveness of the FSA‐Shared_Nadam head pose estimation algorithm and the proposed face quality assessment approach.

In order to estimate head pose precisely in real time with computer vision technology, an enhanced framework using depth data and random regression forest is implemented for head pose estimation. This framework bases on head position and direction point recognition to accomplish head pose estimation. When training random forest, a decision function derived from Haar-like features is used as the binary test and this test uses some data features like Gaussian Curvature and Mean Curvature besides depth value and normal vector. We also generate a large training dataset of range images of heads by virtual structured light scanning. All votes of patches are filtered by clustering and mean shift, and then mean of them are used to estimate position of feature points. Performance evaluation shows accurate pose estimation (success rate above 90%) when running at real-time speed.

Head pose can indicate the eye-gaze direction and face toward which is an important part of human motion estimation and understanding. Due to physical factors of the camera, shooting environment, as well as the appearance change of humanity, the head pose estimation becomes a challenging task. Synchronization sub manifold embedding can find the internal structure of nonlinear data for nonlinear dimensionality reduction and random regression forests can make the nonlinear function mapping for getting the right head pose. In this paper, the advantages of these two algorithms are combined with a method for solving the head pose estimation. Data collection step, the depth data come from the 3D sensor; and training data step, the data is using the local linear structure for label and using a statistical model for synchronization pose samples. Meanwhile the experimental results on a publicly available database prove that the proposed algorithm can achieve state-of-the-art performance while the current estimate has a faster speed and higher robustness when large range of pose changes and outperforms existing.

Age estimation from human faces is an important yet challenging task in computer vision because of the large differences between physical age and apparent age. Due to the differences including races, genders, and other factors, the performance of a learning method for this task strongly depends on the training data. Although many inspiring works have focused on the age estimation of a single human face through deep learning, the existing methods still have lower performance when dealing with faces in videos because of the differences in head pose between frames, which can lead to greatly different results. In this paper, a combined system of age estimation and head pose estimation is proposed to improve the performance of age estimation from faces in videos. We use deep regression forests (DRFs) to estimate the age of facial images, while a multiloss convolutional neural network is also utilized to estimate the head pose. Accordingly, we estimate the age of faces only for head poses within a set degree threshold to enable value refinement. First, we divided the images in the Cross-Age Celebrity Dataset (CACD) and the Asian Face Age Dataset (AFAD) according to the estimated head pose degrees and generated separate age estimates for images with different poses. The experimental results showed that the accuracy of age estimation from frontal facial images was better than that for faces at different angles, thus demonstrating the effect of head pose on age estimation. Further experiments were conducted on several videos to estimate the age of the same person with his or her face at different angles, and the results show that our proposed combined system can provide more precise and reliable age estimates than a system without head pose estimation.

Estimating head pose is a hot topic in facial behavior analysis and understanding. Most of the existing methods called two-stage method take head pose estimation and face detection as two separate tasks. In general, independent face boxes need to be proposed before head pose estimation. Such scheme is inefficient and has poor robustness. The existing estimation methods for head pose is lack of effective anti-noise design. In this paper, we propose a multi-task deep learning method, which integrate face detection and pose estimation together. Three kind of anti-interference strategy are proposed. Compared with the existing two-stage method, the proposed method can be performed with less consumption of resource. Benefited from the complementary characteristics of multi task joint learning, our proposed has higher accuracy. Experiments on several public datasets fully show that the attitude angle estimation error accuracy of our one stage algorithm reaches 1.96° (MAE). It is better than the existing state of the art method. The speed is twice as fast as that of the two-stage method.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

This paper introduces our head pose estimation system that localizes nose-tip of the faces and estimate head poses in studio quality pictures. After the nose-tip in the training data are manually labeled, the appearance variation caused by head pose changes is characterized by tensor model. Given images with unknown head pose and nose-tip location, the nose-tip of the face is localized in a coarse-to-fine fashion, and the head pose is estimated simultaneously by the head pose tensor model. The image patches at the localized nose tips are then cropped and sent to two other head pose estimators based on LEA and PCA techniques. We evaluated our system on the Pointing'04 head pose image database. With the nose-tip location known, our head pose estimators can achieve 94 - 96% head pose classification accuracy(within ±15°). With nose-tip unknown, we achieves 85% nose-tip localization accuracy (within 3 pixels from the ground truth), and 81 - 84% head pose classification accuracy(within ±15°).

Recently, soft biometric trait classification has been receiving more attention in the computer vision community due to its wide range of possible application areas. Most approaches in the literature have focused on trait classification in controlled environments, due to the challenges presented by real-world environments, i.e. arbitrary facial expressions, arbitrary partial occlusions, arbitrary and nonuniform illumination conditions and arbitrary background clutter. In recent years, trait classification has started to be applied to real-world environments, with some success. However, the focus has been on estimation from single images or video frames, without leveraging the temporal information available in the entire video sequence. In addition, a fixed set of features are usually used for trait classification without any consideration of possible changes in the facial features due to head pose changes. In this paper, we propose a temporal, probabilistic framework first to robustly estimate continuous head pose angles from real-world videos, and then use this pose estimate to decide on the appropriate set of frames and features to use in a temporal fusion scheme for soft biometric trait classification. Experiments performed on large, real-world video sequences show that our head pose estimator outperforms the current state-of-the-art head pose approaches (by up to 51%), whereas our head pose conditioned biometric trait classifier (for the case of gender classification) outperforms the current state-of-the-art approaches (by up to 31%).

Most surveillance cameras have a wide-angle field of view and are situated unobtrusively at overhead positions. For this type of application, head pose estimation is very challenging because of the limitations of the quality and resolution of the incoming data. In addition, even though the absolute head pose is constant, the head pose in camera view changes depending upon the location of head with respect the camera. We present a solution to estimate absolute coarse head pose for wide-angle overhead cameras by integrating 3D head position and pose information. The work involves image-based learning, pose correction based on 3D position, and real-time multicamera integration of low-resolution imagery. The system can be applied to an active face catalogger to obtain the best view of the face for surveillance, to customer relationship management to record behavior in retail stores or to virtual reality as an input device.

Although the ability to estimate the face pose and recognise its identity are common human abilities, they are still a challenge in computer vision context. In this study, the authors aim to overcome these difficulties by learning a non‐linear tensor‐based model based on multi‐linear decomposition. Proposed model maps the high‐dimensional image space into low‐dimensional pose manifold. For preserving the actual distance along the manifold shape, a graph‐based distance measure is proposed. Also, to compensate for the limited number of training poses, mirrored images are added to training ones to improve the recognition accuracy. For performance evaluation of the proposed method, experiments are run on three famous face databases using three different manifold shapes and two different distance measures. Eight training data modes are chosen such that the influential parameters are studied comprehensively. The obtained results confirm the effectiveness of proposed model in achieving high accuracy in pose estimation and multi‐view face recognition, even with different training poses for different identities.

Head pose estimation has attracted many research interest in recent years. With the advent of deep learning, it is possible to predict the head pose accurately from the RGB images without the help of facial landmarks or depth information. However, existing head pose datasets often lack large pose head images, which extremely limits the development of head pose estimation algorithms. In this paper, we build the largescale head pose dataset (LHPD) including more than 140,000 images with the diverse and accurate head poses. The LHPD dataset includes the head images recorded from different shooting angles between the camera and the human body for the first time, which greatly expands the range of head pose compared to previous datasets. Therefore, the range of head pose can cover +/-90° for each Euler angle. The accurate and reliable head pose annotation is labeled by the motion capture system and careful calibration procedures. We then propose a head pose estimation method through fine-tuning the ResNet on the LHPD dataset when using the Euclidean distance of quaternions as the loss function. The results show that our method achieves better performance than current state-of-theart algorithms.

One of the challenging problems encountered by face recognition is the difficulty in tackling pose variations, where fast and reliable head pose estimation is an essential step. In this paper, based on the correlation filter technique, a novel feature extraction framework, i.e., directional correlation filter set (DCFS), is developed for robust head pose estimation. In this framework, a principal optimal tradeoff filter (called POTF) is designed in the feature subspace obtained according to principal component analysis (PCA). Compared with the traditional methods that rely on the exact localization of facial feature points, our proposed method exploits the 1D frequency domain of the training data by using the correlation filter technique, which can capture the high-order statistics of a face for effective head pose estimation. Experimental results on several public face databases with large pose variations, including PIE, HPI, and UMIST, show the promising performance obtained by the proposed method on head pose estimation.KeywordsHead pose estimationfeature extractioncorrelation filter