Collaborative acquisition of multi-view face images in real-time using a wireless camera network

Face detection is important in many algorithms in the areas of machine object recognition and pattern recognition. The kaleidoscope of applications for face detection extends across automatic image and home video content annotation, face-image stabilisation and face recognition systems. By using information derived from colour, luminosity and frequency the face detection algorithm proposed in this paper aims to determine the location of multiple frontal and non-frontal faces in compressed MPEG-1, MPEG-2 video or JPEG image content. The described algorithm requires only low computational resources on CE devices and offers at one and the same time extremely high detection rates.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

Many Facial expression recognition methods have achieved great success, but they only considered front facial images or facial images close to the front. Besides, unlike in-the-laboratory datasets, the facial images in the real world (or in the wild) are without lighting and pose control, so that it is a big challenge to recognize these facial expressions. In this paper, the PIDViT (i.e., Pose-Invariant Distilled Vision Transformer) using the teacher-student model for the probability distributions of facial expressions of frontal and multi-pose faces was proposed and solved the pose variance and occlusion issues on expression recognition. First, the multi-pose face dataset FairFace-3D was built from the original FairFace and then used to train pose-invariance on the PIDViT. The PIDViT was trained in two stages; stage 1 is to train the PIDViT to achieve the consistency of facial expressions between frontal faces and multi-pose faces, and stage 2 is to use the student model pre-trained in stage 1 and train facial expressions further on target datasets. Finally, comprehensive experiments were conducted on three in the wild facial expression datasets, and the results validates the generalization of the PIDViT and its superiority over most state-of-the-art models.

The novel eye-based human-computer interaction (HCI) system aims to provide people, especially, disabled persons, a new way of communication with surroundings. It adopts a series of continual eye movements as input to perform simple control activities. Identification of eye movements is the crucial technology in these eye-based HCI systems. At present, researches on eye movement identification mainly focus on frontal face images. In fact, acquisition of non-frontal face images is more reasonable in real applications. In this paper, we discuss the identification process of eye movements from non-frontal face images. Firstly, the original head-shoulder images of 0°---±60° azimuths are sampled without any auxiliary light source. Secondly, the non-frontal face region is detected by using the Adaboost cascade classifiers. After that, we roughly extract eye windows by the integral projection function. Then, we propose a new method to calculate the x ? y coordinates of the pupil center point by searching the minimal intensity value in the eye windows. According to the trajectory of the pupil center points, different eye movements (eye moving left, right, up or down) are successfully identified. A set of experiments is presented.

In this paper, we present a system to monitor a subject's Visual Focus of Attention (VFOA) based on his/her head poses. The system first detects faces from video images and determines if the detected face is a frontal or profile face. If a frontal face is detected, the system further estimates the head pose from the face image. Instead of estimating accurate head poses through detection or tracking methods, we formulate the problem as a classification problem and classify the head pose into one of a predefined number of poses using a local discriminant projection (LDP) method. The LDP method uses two graphs for the modeling the head pose embedding, one is the nearest native neighbor graph, the other is the nearest invader graph. We evaluate the LDP method in CAS-PEAL Database with 21 head poses and a realistic data set with 9 poses collected from our application scenario. The experimental results indicate that our approach outperforms other methods. We describe the implementation of the system with an application in monitoring customers' VFOA in a display window that exhibits merchandise in a shop. The system can be used to index and retrieve information for customer analysis.

It is well known that frontal face is much easier to be recognized than pose-variant face for both human and machine perception. However, it is not easy to acquire a frontal face in real-world video surveillance. This paper proposes a method to synthetize a frontal face for recognition in video surveillance scene, which is based on Conditional Generative Adversarial Networks (cGAN) with input of multiple pose-variant faces from a video. Experimental results show that the proposed approach can generate suitable frontal faces and improve face recognition by around 20% on a dataset of 43276 face images from 19 persons, collected from the real-world video surveillance scene. The effectiveness of multiple frames against single frame as input is demonstrated. Moreover, we investigate the generator with different depth for synthetizing frontal faces, in which an up-down sampling trick is designed for synthetizing higher quality frontal face images and boosts the performance of the generator.

How do representations of the face in portraits relate to the natural face, and how does the aesthetics of portraits relate to the aesthetics of faces in photographs? Here we investigate these questions with regard to the frontal face. Frontal faces are of particular interest because they are by far the most commonly studied type of face image in psychology, yet frontal portraits have been little studied by psychologists. Using behavioral and statistical tests, we show that artistic representations of frontal female faces have representational properties that broadly match those of the natural face, but we also find properties unique to artworks. We report that, as with frontal faces, frontal portraits show norm-based coding properties with respect to preference: averaged portraits become more attractive in proportion to the number of portraits averaged together. However, averaged photographs of faces are preferred to averaged portraits, suggesting that faces in portraits and photographed faces show basic differences in aesthetics. Consistent with this notion, we found that average face width and height ratios in an extended sample of frontal female portraits were significantly different from those for photographed faces. This indicates that portraits on average are not faithful representations of the typical structure of the face. In a behavioral experiment where we manipulated the structural ratios in portraits, we found that the preferred width and height ratios were significantly different from those preferred in photographed faces, and that the preferred ratios for portraits were closer to the average ratios of the portrait sample. We evaluate a variety of possible causes of the observed differences. We conclude that despite the demonstrated differences between artistic representations and natural faces, fundamental properties of natural faces are preserved in artistic representations of the face.

Face recognition in the SWIR band when using single sensor multi-wavelength imaging systems

Face frontalization is one way to overcome the pose variation problem, which simplifies multi-view recognition into one canonical-view recognition. This paper presents a multi-task learning approach based on the generative adversarial network (GAN) that learns the emotion-preserving representations in the face frontalization framework. Taking advantage of adversarial relationship between the generator and the discriminator in GAN, the generator can frontalize input non-frontal face images into frontal face images while preserving the identity and expression characteristics; in the meantime, it can employ the learnt emotion-preserving representations to predict the expression class label from the input face. The proposed network is optimized by combining both synthesis and classification objective functions to make the learnt representations generative and discriminative simultaneously. Experimental results demonstrate that the proposed face frontalization system is very effective for expression recognition with large head pose variations.

In this paper we present a new scheme for detection and tracking of specific objects in a knowledge-based framework. The scheme uses a supervised learning method: support vector machines. Both problems, detection and tracking, are solved by a common approach: objects are located in video sequences by a SVM classifier. They are next tracked along the time by a SVM tracker with complete 6 parameters affine model. The method is applied in a video surveillance application for detection and tracking of frontal view faces. Real time application constraints are met by reduction of support vector set.

Image set based face recognition has recently become a popular topic as it has better performance than single image based face recognition. However, preprocessing is needed to remove the effects of some adverse conditions such as different pose angles, illumination, and expression differences within the set. One of the most effective preprocessing to improve the face recognition rate is face frontalization. Face frontalization is defined as the artificial acquisition of a face image with a different pose angle to a frontal pose. It has been observed that this process increases the face recognition performance. In this paper, image set based face recognition was performed by applying face frontalization to all images in the sets. Firstly, the faces in the IJBA database were frontalized by using the Rotate and Render hybrid frontalization method, which is based on a Three-Dimensional and Generative Adversial Network. Then, discriminative convex classifier is used for set based face recognition. In face recognition experiments, when the frontalized IJBA database and its non-frontalized version were compared, it was observed that the accuracy of face recognition increased with the frontalized face images.

Face detection is a common computer technology being used in human identification applications. It can also refer to the process of locating human faces in a visual scene. Face detection is a branched field of object detection where all objects in an image are detected including several classes like cars, trees, humans… etc. Also face detection problems branch into a lot of cases, some focus on frontal faces, others focus on side pose and so on. In this paper, a new face detection method based on Bilinear Interpolation image zooming method and image enhancement by Adaptive Histogram Equalization (AHE) method is proposed. The new method gives an encouraging results for crowded human images. By comparing the proposed method with the Viola-Jones algorithm, face detector using the cascade object detector, which supported in MATLAB, the new method gives excellent results in detecting human faces with different resolutions, poses and sizes. It succeeds in detecting most of the human faces in the tested images regardless of image sizes. The new method is tested on several images in Pratheepan dataset with crowded humans. Also, I tested the new method on many images collected from the Internet, whose can be classified as crowded human images. Experimental results show that the proposed Ad_L_Hist method is more efficient in detecting human faces in crowded human images.

Traditional frontal face image synthesis based on the ℓ 1 ‐penalty has achieved remarkable success. However, the ℓ 1 ‐penalty on reconstruction coefficients has the drawback of instability when processing high‐dimensional data (e.g. a facial image including hundreds of pixels). Moreover, the traditional ℓ 1 ‐penalty‐based method requires consistency between the corresponding patches in frontal and profile faces, which is hard to guarantee due to self‐occlusion. To overcome the instability problem of the traditional method, an extension of the ℓ 1 ‐penalty‐based frontal face synthesis method, which benefits from the nature of the elastic net, is presented. 3 addition, to enhance the aforementioned consistency, a neighbourhood consistency penalty is imposed onto the reconstruction coefficients using the connected neighbour patches of the current patch. Furthermore, to ensure the synthesised result faithfully approximates the ground truth, a sparse neighbour selection strategy is introduced for finding related neighbours adaptively. Experimental results demonstrate the superiority of the proposed method over some state‐of‐the‐art methods in both visual and quantitative comparisons.

In a video analytics for audience measurement system, dwell time, gaze, and opportunity-to-see statistics are required most of the time. To generate these statistics, more than one face detector is used in order to capture both profile and frontal faces. In this paper, we present a novel approach for face detection in video analytics. The assumption is that; the face occurrences are limited in such systems and one classifier is able to capture all of these occurrences. By using MB-LBP for feature extraction and Gentle Boost for statistical learning we trained a classifier which is able to detect both profiles and frontal faces with more than 80 % success rate. The proposed system also uses an efficient scanning algorithm and achieves 60 fps on a 720p video.

HifiDiff: High-fidelity diffusion model for face hallucination from tiny non-frontal faces