On-the-move heterogeneous face recognition in frequency and spatial domain using sparse representation

Heterogeneous face recognition (HFR) aims to match face images across different imaging domains such as visible-to-infrared and visible-to-thermal. Recently, the increasing utility of nonvisible imaging has increased the application prospects of HFR in areas such as biometrics, security, and surveillance. HFR is a challenging variate of face recognition due to the differences between different imaging domains. While the current research has proposed image preprocessing, feature extraction, or common subspace projection for HFR, the optimization of these multi-stage methods is a challenging task as each step needs to be optimized separately and the performance error accumulates over each stage. In this paper, we propose a unified end-to-end Cross-Modality Discriminator Network (CMDN) for HFR. The proposed network uses a Deep Relational Discriminator module to learn deep feature relations for cross-domain face matching. Simultaneously, the CMDN is used to extract modality-independent embedding vectors for face images. The CMDN parameters are optimized using a novel Unit-Class Loss that shows higher stability and accuracy over other popular metric-learning loss functions. The experimental results on five popular HFR datasets demonstrate that the proposed method achieves significant improvement over the existing state-of-the-art methods.

Multi-level dynamic error coding for face recognition with a contaminated single sample per person

The issue of single sample per person (SSPP) face recognition has attracted more and more attention in recent years. Patch/local-based algorithm is one of the most popular categories to address the issue, as patch/local features are robust to face image variations. However, the global discriminative information is ignored in patch/local-based algorithm, which is crucial to recognize the nondiscriminative region of face images. To make the best of the advantage of both local information and global information, a novel two-layer local-to-global feature learning framework is proposed to address SSPP face recognition. In the first layer, the objective-oriented local features are learned by a patch-based fuzzy rough set feature selection strategy. The obtained local features are not only robust to the image variations, but also usable to preserve the discrimination ability of original patches. Global structural information is extracted from local features by a sparse autoencoder in the second layer, which reduces the negative effect of nondiscriminative regions. Besides, the proposed framework is a shallow network, which avoids the over-fitting caused by using multilayer network to address SSPP problem. The experimental results have shown that the proposed local-to-global feature learning framework can achieve superior performance than other state-of-the-art feature learning algorithms for SSPP face recognition.

Joint and collaborative representation with local adaptive convolution feature for face recognition with single sample per person

Identifying the type of modalities of the query image which can be of types visual, NIR, digital camera, web camera etc. have been assumed to be available before face matching. This leads to a major drawback in achieving fully automated heterogeneous face recognition as real world scenarios cannot be reflected. Therefore, modality identification is an important component of the heterogeneous face recognition system which is being overlooked by majority of the state-of-the-art methods. This component should be given similar attention when comparing with other face recognition modules identifying pose, gesture, camera source etc. In this paper inspired from sensor pattern noise (SPN) estimation based approaches, a novel image sharpening based modality pattern noise technique is proposed for modality identification. The proposed system has been evaluated on three challenging benchmarks of heterogeneous face databases. The proposed technique has produced outstanding results and will open new avenues of research for automated HFR methods in future.

In this paper we propose a novel face hallucination algorithm to synthesize a high-resolution face image from several low-resolution input face images. As described in Liu et al. [8]'s work, face hallucination uses two models: a global parametric model which synthesizes global face shapes from eigenfaces, and a local parametric model which enhances the local high frequency details. We follow a similar process to develop a robust face hallucination algorithm. Firstly, we obtain eigenfaces from a number of low resolution face images extracted from a video sequence using a face tracking algorithm. Then we compute the difference between the interpolated low-resolution face and the mean face, and use this difference face as the query to retrieve approximate sparse eigenfaces representation. The eigenfaces are combined using the coefficients obtained from sparse representation and added into the interpolated low-resolution face. In this way, the global shape of the interpolated low resolution face can be successfully enhanced. Secondly, we improve the example-based super-resolution method [7] for local high frequency information enhancement. Our proposed algorithm uses the Approximate Nearest Neighbors (ANN) search method to find a number of nearest neighbors for a stack of queries, instead of finding the exact match for each low frequency patch as presented in [7]. Median filtering is used to remove the noise from the nearest neighbors in order to enhance the signal. Our proposed algorithm uses sparse representation and the ANN method to enhance both global face shape and local high frequency information while greatly improving the processing speed, as confirmed empirically.

Improved sparse representation with low-rank representation for robust face recognition

Heterogeneous face recognition (HFR) refers to matching face images acquired from different sources (i.e., different sensors or different wavelengths) for identification. HFR plays an important role in both biometrics research and industry. In spite of promising progresses achieved in recent years, HFR is still a challenging problem due to the difficulty to represent two heterogeneous images in a homogeneous manner. Existing HFR methods either represent an image ignoring the spatial information, or rely on a transformation procedure which complicates the recognition task. Considering these problems, we propose a novel graphical representation based HFR method (G-HFR) in this paper. Markov networks are employed to represent heterogeneous image patches separately, which takes the spatial compatibility between neighboring image patches into consideration. A coupled representation similarity metric (CRSM) is designed to measure the similarity between obtained graphical representations. Extensive experiments conducted on multiple HFR scenarios (viewed sketch, forensic sketch, near infrared image, and thermal infrared image) show that the proposed method outperforms state-of-the-art methods.

Real-world face recognition systems often have to face the single sample per person (SSPP) problem, that is, only a single training sample for each person is enrolled in the database. In this case, many of the popular face recognition methods fail to work well due to the inability to learn the discriminatory information specific to the persons to be identified. To address this problem, in this paper, we propose an Adaptive Generic Learning (AGL) method, which adapts a generic discriminant model to better distinguish the persons with single face sample. As a specific implementation of the AGL, a Coupled Linear Representation (CLR) algorithm is proposed to infer, based on the generic training set, the within-class scatter matrix and the class mean of each person given its single enrolled sample. Thus, the traditional Fisher's Linear Discriminant (FLD) can be applied to SSPP task. Experiments on the FERET and a challenging passport face database show that the proposed method can achieve better results compared with other common solutions to the SSPP problem.

Existing heterogeneous face synthesis (HFS) methods focus on performing accurate image-to-image translation across domains, while they cannot effectively remove the nuisance facial variations such as poses, expressions or occlusions. To address such challenges, this paper studies a new practical heterogeneous prototype learning (HPL) problem. To be specific, given a face image contaminated by facial variations from a source domain, HPL aims to reconstruct the variation-free prototype in a specified target domain. To tackle HPL, we propose a unified and end-to-end framework named bidirectional heterogeneous prototype learning (BHPL). As a bidirectional learning framework, BHPL is able to simultaneously reconstruct the heterogeneous prototypes across <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">source-to-target</i> as well as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">target-to-source</i> domains. Furthermore, BHPL is capable of learning the identity prototype features for the contaminated face images from both source and target domains in order to perform robust heterogeneous face recognition. BHPL consists of an encoder-decoder structural generator and two dual-task discriminators, which play an adversarial game such that the generator learns the identity prototype feature and generates the cross-domain identity-preserved prototype for each input face image from both domains, and the discriminators accurately predict face identity and distinguish real versus fake prototypes. Empirically studies on multiple heterogeneous face datasets containing facial variations demonstrate the effectiveness of BHPL.

Age estimation using face images has been widely employed across various fields. Because the characteristics of face images usually vary greatly depending on race, camera type, lighting, and other environmental factors, the recognition ability of untrained heterogeneous face image databases is not accurate by previous methods. Therefore, various attempts have been made where different heterogeneous databases were combined to enable training; however, the training time is extended and diverse environmental variables in databases cannot be sufficiently trained. To address these issues, this study proposes modified cycle-consistent generative adversarial network (CycleGAN) that generates an even distribution of heterogeneous face data, and an age-estimation method that is effective for heterogeneous data based on comparative CNN for age estimation (CCNNAE). In addition, we propose the method of reducing the errors caused by image transformation in modified CycleGAN through adaptive selection between transformed and original face images for the input to CCNNAE, which is based on age similarity between the transformed face image and the original face image. Experiments with two open databases, MORPH and MegaAge databases showed that our method outperformed the state-of-the art methods.

Spatial domain facial recognition Modular IMage Principal Component Analysis (MIMPCA) has an improved recognition rate compared to the conventional PCA. In the MPCA, face images are divided into smaller sub-images and the PCA approach is applied to each of these sub-images. In this work, the Transform Domain implementation of MPCA is presented. The facial image has two representations. The Two Dimensional MPCA (TD - 2D - MPCA) and the Diagonal matrix MPCA (TD - Dia - MPCA). The sub-images are processed using both non-overlapping and overlapping windows. All the test results, for noise free and noisy images, using ORL, Yale and FERET databases achieved; 99.5%, 99.58% and 97.42% recognition accuracy respectively. Transform Domain implementations yield, computational and storage savings of at least 75% and 99.98%, respectively, compared to spatial domain. Sample results are given.

This paper describes in detail about important 2D and 3D face and video face databases. The face and video face databases discussed here includes: Indian Institute of Technology Kanpur (IIT-K) database, Makeup dataset, University of North Carolina Wilmington Craniofacial Morphology (MORPH) non-commercial face database, Surveillance face (SCface) database, YouTube face database and University of Mialno Bicocca 3D face database (UMBDB). The facial image captured may be colour or gray, with or without makeup, same pose or different poses, occluded or not occluded, neutral expression or non-neutral expression (angry, sad, laughter, smile, sleepy), different illumination conditions, in the presence of visible light or Infrared captured. The subject may be Asian or African or American or European. For above said reasons, different face and video face databases developed in the past 20 years and few important databases are analyzed here.

In this paper, a person-specific facial expression recognition method which is based on Personal Facial Expression Space (PFES) is presented. The multidimensional scaling maps facial images as points in lower dimensions in PFES. It reflects personality of facial expressions as it is based on the peak instant of facial expression images of a specific person. In constructing PFES for a person, his/her whole normalized facial image is considered as a single pattern without block segmentation and differences of 2-D DCT coefficients from neutral facial image of the same person are used as features. Therefore, in the early part of the paper, separation characteristics of facial expressions in the frequency domain are analyzed using a still facial image database which consists of neutral, smile, anger, surprise and sadness facial images for each of 60 Japanese males (300 facial images). Results show that facial expression categories are well separated in the low frequency domain. PFES is constructed using multidimensional scaling by taking these low frequency domain of differences of 2-D DCT coefficients as features. On the PFES, trajectory of a facial image sequence of a person can be calculated in real time. Based on this trajectory, facial expressions can be recognized. Experimental results show the effectiveness of this method.© (2000) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Most face recognition methods employ single-bit binary descriptors for face representation. The information from these methods is lost in the process of quantization from real-valued descriptors to binary descriptors, which greatly limits their robustness for face recognition. In this study, we propose a novel weighted feature histogram (WFH) method of multi-scale local patches using multi-bit binary descriptors for face recognition. First, to obtain multi-scale information of the face image, the local patches are extracted using a multi-scale local patch generation (MSLPG) method. Second, with the goal of reducing the quantization information loss of binary descriptors, a novel multi-bit local binary descriptor learning (MBLBDL) method is proposed to extract multi-bit local binary descriptors (MBLBDs). In MBLBDL, a learned mapping matrix and novel multi-bit coding rules are employed to project pixel difference vectors (PDVs) into the MBLBDs in each local patch. Finally, a novel robust weight learning (RWL) method is proposed to learn a set of robust weights for each patch to integrate the MBLBDs into the final face representation. In RWL, a codebook is first constructed by clustering MBLBDs on each local patch to extract a feature histogram. Then, considering that different parts of the face have different degrees of robustness to local changes, a set of weights is learned to concatenate the feature histograms of all local patches into the final representation of a face image. In addition, to further improve the performance for heterogeneous face recognition, a coupled WFH (C-WFH) method is proposed. C-WFH maintains the similarity of the corresponding MBLBDs and feature histograms for a pair of heterogeneous face images by means of a novel coupled feature learning (CFL) method to reduce the modality gap. A series of experiments are conducted on widely used face datasets to analyze the performance of WFH and C-WFH. Extensive experimental results show that WFH and C-WFH outperform state-of-the-art face recognition methods.