Frontal Face Generation from Multiple Pose-Variant Faces with CGAN in Real-World Surveillance Scene

Many Facial expression recognition methods have achieved great success, but they only considered front facial images or facial images close to the front. Besides, unlike in-the-laboratory datasets, the facial images in the real world (or in the wild) are without lighting and pose control, so that it is a big challenge to recognize these facial expressions. In this paper, the PIDViT (i.e., Pose-Invariant Distilled Vision Transformer) using the teacher-student model for the probability distributions of facial expressions of frontal and multi-pose faces was proposed and solved the pose variance and occlusion issues on expression recognition. First, the multi-pose face dataset FairFace-3D was built from the original FairFace and then used to train pose-invariance on the PIDViT. The PIDViT was trained in two stages; stage 1 is to train the PIDViT to achieve the consistency of facial expressions between frontal faces and multi-pose faces, and stage 2 is to use the student model pre-trained in stage 1 and train facial expressions further on target datasets. Finally, comprehensive experiments were conducted on three in the wild facial expression datasets, and the results validates the generalization of the PIDViT and its superiority over most state-of-the-art models.

Background information is crucial for many video surveillance applications such as object detection and scene understanding. In this paper, we present a novel pixel-to-model (P2M) paradigm for background modeling and restoration in surveillance scenes. In particular, the proposed approach models the background with a set of context features for each pixel, which are compressively sensed from local patches. We determine whether a pixel belongs to the background according to the minimum P2M distance, which measures the similarity between the pixel and its background model in the space of compressive local descriptors. The pixel feature descriptors of the background model are properly updated with respect to the minimum P2M distance. Meanwhile, the neighboring background model will be renewed according to the maximum P2M distance to handle ghost holes. The P2M distance plays an important role of background reliability in the 3-D spatial–temporal domain of surveillance videos, leading to the robust background model and recovered background videos. We applied the proposed P2M distance for foreground detection and background restoration on synthetic and real-world surveillance videos. Experimental results show that the proposed P2M approach outperforms the state-of-the-art approaches both in indoor and outdoor surveillance scenes.

In this paper, we present a system to monitor a subject's Visual Focus of Attention (VFOA) based on his/her head poses. The system first detects faces from video images and determines if the detected face is a frontal or profile face. If a frontal face is detected, the system further estimates the head pose from the face image. Instead of estimating accurate head poses through detection or tracking methods, we formulate the problem as a classification problem and classify the head pose into one of a predefined number of poses using a local discriminant projection (LDP) method. The LDP method uses two graphs for the modeling the head pose embedding, one is the nearest native neighbor graph, the other is the nearest invader graph. We evaluate the LDP method in CAS-PEAL Database with 21 head poses and a realistic data set with 9 poses collected from our application scenario. The experimental results indicate that our approach outperforms other methods. We describe the implementation of the system with an application in monitoring customers' VFOA in a display window that exhibits merchandise in a shop. The system can be used to index and retrieve information for customer analysis.

In order to support real-time face recognition using a wireless camera network, we design a data acquisition service to quickly and reliably acquire face images of human subjects from multiple views and to simultaneously index each acquired image into its corresponding pose. In comparison with detection of frontal faces, the detection of non-frontal faces with unknown pose is a much more challenging problem that involves significant image processing. In this paper, we describe a collaborative approach in which multi-view camera geometry and inter-camera communication is utilized at run time to significantly reduce the required processing time. By doing so, we are able to achieve a high capture rate for both frontal and non-frontal faces and at the same time maintain a high detection accuracy. We implement our face acquisition system on a 1.6 GHz Intel Atom Processor based embedded camera network and show that we can reliably acquire frontal faces at 11 fps and non-frontal faces at 10 fps on images captured at a resolution of 640 × 480 pixels.

How do representations of the face in portraits relate to the natural face, and how does the aesthetics of portraits relate to the aesthetics of faces in photographs? Here we investigate these questions with regard to the frontal face. Frontal faces are of particular interest because they are by far the most commonly studied type of face image in psychology, yet frontal portraits have been little studied by psychologists. Using behavioral and statistical tests, we show that artistic representations of frontal female faces have representational properties that broadly match those of the natural face, but we also find properties unique to artworks. We report that, as with frontal faces, frontal portraits show norm-based coding properties with respect to preference: averaged portraits become more attractive in proportion to the number of portraits averaged together. However, averaged photographs of faces are preferred to averaged portraits, suggesting that faces in portraits and photographed faces show basic differences in aesthetics. Consistent with this notion, we found that average face width and height ratios in an extended sample of frontal female portraits were significantly different from those for photographed faces. This indicates that portraits on average are not faithful representations of the typical structure of the face. In a behavioral experiment where we manipulated the structural ratios in portraits, we found that the preferred width and height ratios were significantly different from those preferred in photographed faces, and that the preferred ratios for portraits were closer to the average ratios of the portrait sample. We evaluate a variety of possible causes of the observed differences. We conclude that despite the demonstrated differences between artistic representations and natural faces, fundamental properties of natural faces are preserved in artistic representations of the face.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

Image set based face recognition has recently become a popular topic as it has better performance than single image based face recognition. However, preprocessing is needed to remove the effects of some adverse conditions such as different pose angles, illumination, and expression differences within the set. One of the most effective preprocessing to improve the face recognition rate is face frontalization. Face frontalization is defined as the artificial acquisition of a face image with a different pose angle to a frontal pose. It has been observed that this process increases the face recognition performance. In this paper, image set based face recognition was performed by applying face frontalization to all images in the sets. Firstly, the faces in the IJBA database were frontalized by using the Rotate and Render hybrid frontalization method, which is based on a Three-Dimensional and Generative Adversial Network. Then, discriminative convex classifier is used for set based face recognition. In face recognition experiments, when the frontalized IJBA database and its non-frontalized version were compared, it was observed that the accuracy of face recognition increased with the frontalized face images.

Reconstruction of a 3D face from a single 2D face image can be substantially useful in many areas such as security, forensic, 3D animation, and motion capture. Most 3D face reconstruction techniques require multiple 2D face images taken at different views in order to estimate the depth of each face component which is one of the most crucial parameters in 3D face reconstruction. We propose a new method that is capable of estimating the depth order of a 2D face image in various illumination conditions. The proposed method uses Hillcrest-Valley classification with adaptive mean filter and Otsu threshold value selection to estimate the depth order of an image. The proposed method was evaluated on the YaleB face database consists of 10 individual frontal faces for the total of 576 face images and the experimental results reveal that the proposed method is robust and it can estimate the depth order of a frontal face up to as many as 50 layers regardless of lighting condition.

Traditional frontal face image synthesis based on the ℓ 1 ‐penalty has achieved remarkable success. However, the ℓ 1 ‐penalty on reconstruction coefficients has the drawback of instability when processing high‐dimensional data (e.g. a facial image including hundreds of pixels). Moreover, the traditional ℓ 1 ‐penalty‐based method requires consistency between the corresponding patches in frontal and profile faces, which is hard to guarantee due to self‐occlusion. To overcome the instability problem of the traditional method, an extension of the ℓ 1 ‐penalty‐based frontal face synthesis method, which benefits from the nature of the elastic net, is presented. 3 addition, to enhance the aforementioned consistency, a neighbourhood consistency penalty is imposed onto the reconstruction coefficients using the connected neighbour patches of the current patch. Furthermore, to ensure the synthesised result faithfully approximates the ground truth, a sparse neighbour selection strategy is introduced for finding related neighbours adaptively. Experimental results demonstrate the superiority of the proposed method over some state‐of‐the‐art methods in both visual and quantitative comparisons.

Video surveillance systems play a crucial role in ensuring public safety and security. However, the traditional methods of surveillance often fall short in effectively identifying individuals, particularly in crowded or dynamic environments. This research addresses the limitations of conventional video surveillance by proposing a machine learning-based facial recognition system. The increasing demand for robust security measures necessitates the development of advanced technologies in video surveillance. Facial recognition has emerged as a promising solution, but existing systems struggle with accuracy and efficiency. This research aims to bridge these gaps by leveraging machine learning techniques for facial recognition in video surveillance. Conventional video surveillance struggles with accurate and rapid identification of individuals, leading to potential security lapses. This research addresses the challenge of enhancing facial recognition accuracy in real-time video feeds, especially in scenarios with varying lighting conditions and occlusions. While facial recognition has gained traction, there is a significant research gap in the implementation of machine learning algorithms tailored for video surveillance. This study aims to fill this void by proposing a novel methodology that combines deep learning and computer vision techniques for robust facial recognition in dynamic environments. The proposed methodology involves training a deep neural network on a diverse dataset of facial images to enable the model to learn intricate facial features. Additionally, computer vision algorithms will be employed to handle challenges such as occlusions and varying lighting conditions. The model''''s performance will be evaluated using real-world video surveillance data. Preliminary results demonstrate a significant improvement in facial recognition accuracy compared to traditional methods. The machine learning-based system exhibits enhanced performance in challenging scenarios, showcasing its potential for practical implementation in video surveillance systems.

Techniques to extract or understand interactions between moving objects in video is becoming increasingly important as the amount of video increases. Applications in surveillance range from understanding traffic to studying fish schooling behaviour. Because of the massive amount of data, fast, approximate techniques based on statistical models are common. These models connect user annotations (labels) to scenes in a (short) video segment. The connection forms a domain, which associates information about moving objects in scenes with the labels, such as to indicate whether a user considers a particular traffic scene to be "dangerous." Unfortunately a statistical model trained in one domain often yields low precision and recall when applied to another domain because the random variables that explain video content exhibit changing marginal and conditional probability distributions over time (e.g., due to different backgrounds, changes in illumination, shading, and numbers of moving objects). This problem is exacerbated when new domains continuously arise (e.g., in the real-time processing of video) and user annotations are only limited to training data, a common scenario for surveillance video. In this paper, we propose a new, cross-domain technique that reuses labelled content from source domains to improve the prediction of user annotations in a target domain. Our model probabilistically learns how users annotate scenes based on the similarity of target to source domains. Two domains that are similar will share a large number of observable features. We encode the similarity in a covariance matrix, which flexibly allows allows users to set an arbitrary covariance structure between pairs of domains before training the model. Experiments show that our method improves state-of-the-art techniques (SVM and CF) in predicting dangerous scenes in real-world traffic surveillance videos.

Recently many important face recognition systems could deal well with frontal view face images. However few of them work well when there is only one training image per person. We propose an approach to cope with the problem by using 1D discrete hidden Markov model (1D-DHMM). The model training and recognition part were carried out on both vertical and horizontal directions. New way of extracting observations and using observation sequences in recognition is introduced. The Haar wavelet transform was applied to the image to lessen the dimension of the observation vectors. Our experiment results tested on the frontal view AR Face Database show that the proposed method outperforms the PCA, LDA, LFA approaches tested on the same database.

Pose and textural variations are two dominant factors to affect the performance of face recognition. It is widely believed that generating the corresponding frontal face from a face image of an arbitrary pose is an effective step toward improving the recognition performance. In the literature, however, the frontal face is generally recovered by only exploring textural characteristic. In this letter, we propose a two-stream deep correlation network, which incorporates both geometric and textural features for frontal face recovery. Given a face image under an arbitrary pose as input, geometric and textural characteristics are first extracted from two separate streams. The extracted characteristics are then fused through the proposed multiplicative patch correlation layer. These two steps are integrated into one network for end-to-end training and prediction, which is demonstrated effective compared with state-of-the-art methods on the benchmark datasets.

Face frontalization is one way to overcome the pose variation problem, which simplifies multi-view recognition into one canonical-view recognition. This paper presents a multi-task learning approach based on the generative adversarial network (GAN) that learns the emotion-preserving representations in the face frontalization framework. Taking advantage of adversarial relationship between the generator and the discriminator in GAN, the generator can frontalize input non-frontal face images into frontal face images while preserving the identity and expression characteristics; in the meantime, it can employ the learnt emotion-preserving representations to predict the expression class label from the input face. The proposed network is optimized by combining both synthesis and classification objective functions to make the learnt representations generative and discriminative simultaneously. Experimental results demonstrate that the proposed face frontalization system is very effective for expression recognition with large head pose variations.

Robust discriminative nonnegative dictionary learning for occluded face recognition