Face Hallucination: Theory and Practice

Modern face recognition systems extract face representations using deep neural networks (DNNs) and give excellent identification and verification results, when tested on high resolution (HR) images. However, the performance of such an algorithm degrades significantly for low resolution (LR) images. A straight forward solution could be to train a DNN, using simultaneously, high and low resolution face images. This approach yields a definite improvement at lower resolutions but suffers a performance degradation for high resolution images. To overcome this shortcoming, we propose to train a network using both HR and LR images under the guidance of a fixed network, pretrained on HR face images. The guidance is provided by minimising the KL-divergence between the output Softmax probabilities of the pretrained (i.e., Teacher) and trainable (i.e., Student) network as well as by sharing the Softmax weights between the two networks. The resulting solution is tested on down-sampled images from FaceScrub and MegaFace datasets and shows a consistent performance improvement across various resolutions. We also tested our proposed solution on standard LR benchmarks such as TinyFace and SCFace. Our algorithm consistently outperforms the state-of-the-art methods on these datasets, confirming the effectiveness and merits of the proposed method.

This paper proposes a method of synthesizing a high-resolution facial image from a low-resolution facial image based on top-down learning. A face is represented by a linear combination of prototypes of shape and texture. With the shape and texture information about the pixels in an given low-resolution facial image, we can estimate optimal coeficients for a linear combination of prototypes of shape and those of texture by solving least square minimization. Then high-resolution facial image can be synthesized by using the optimal coeficients for linear combination of the high-resolution prototypes. The encouraging results of the proposed method show that our method can be used to increase the performance of the face recognition by applying our method to enhance the low-resolution facial images captured at surveillance systems.KeywordsFace RecognitionFacial ImageInput FaceBicubic InterpolationReference FaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Age estimation using facial images is applicable in various fields, such as age-targeted marketing, analysis of demand and preference for goods, skin care, remote medical service, and age statistics, for describing a specific place. However, if a low-resolution camera is used to capture the images, or facial images are obtained from the subjects standing afar, the resolution of the images is degraded. In such a case, information regarding wrinkles and the texture of the face are lost, and features that are crucial for age estimation cannot be obtained. Existing studies on age estimation did not consider the degradation of resolution but used only high-resolution facial images. To overcome this limitation, this paper proposes a deep convolutional neural network (CNN)-based age estimation method that reconstructs low-resolution facial images as high-resolution images using a conditional generative adversarial network (GAN), and then uses the images as inputs. An experiment is conducted using two open databases (PAL and MORPH databases). The results demonstrate that the proposed method achieves higher accuracy in high-resolution reconstruction and age estimation than the state-of-the art methods.

In this paper we propose a novel face hallucination algorithm to synthesize a high-resolution face image from several low-resolution input face images. As described in Liu et al. [8]'s work, face hallucination uses two models: a global parametric model which synthesizes global face shapes from eigenfaces, and a local parametric model which enhances the local high frequency details. We follow a similar process to develop a robust face hallucination algorithm. Firstly, we obtain eigenfaces from a number of low resolution face images extracted from a video sequence using a face tracking algorithm. Then we compute the difference between the interpolated low-resolution face and the mean face, and use this difference face as the query to retrieve approximate sparse eigenfaces representation. The eigenfaces are combined using the coefficients obtained from sparse representation and added into the interpolated low-resolution face. In this way, the global shape of the interpolated low resolution face can be successfully enhanced. Secondly, we improve the example-based super-resolution method [7] for local high frequency information enhancement. Our proposed algorithm uses the Approximate Nearest Neighbors (ANN) search method to find a number of nearest neighbors for a stack of queries, instead of finding the exact match for each low frequency patch as presented in [7]. Median filtering is used to remove the noise from the nearest neighbors in order to enhance the signal. Our proposed algorithm uses sparse representation and the ANN method to enhance both global face shape and local high frequency information while greatly improving the processing speed, as confirmed empirically.

Face Super-Resolution (FSR) aims to infer High-Resolution (HR) face images from the captured Low-Resolution (LR) face image with the assistance of external information. Existing FSR methods are less effective for the LR face images captured with serious low-quality since the huge imaging/degradation gap caused by the different imaging scenarios (i.e., the complex practical imaging scenario that generates test LR images, the simple manual imaging degradation that generates the training LR images) is not considered in these algorithms. In this paper, we propose an image homogenization strategy via re-expression to solve this problem. In contrast to existing methods, we propose a homogenization projection in LR space and HR space as compensation for the classical LR/HR projection to formulate the FSR in a multi-stage framework. We then develop a re-expression process to bridge the gap between the complex degradation and the simple degradation, which can remove the heterogeneous factors such as serious noise and blur. To further improve the accuracy of the homogenization, we extract the image patch set that is invariant to degradation changes as Robust Neighbor Resources (RNR), with which these two homogenization projections re-express the input LR images and the initial inferred HR images successively. Both quantitative and qualitative results on the public datasets demonstrate the effectiveness of the proposed algorithm against the state-of-the-art methods.

Face recognition has attracted enormous interest from researchers due to its widescale adaptation. However, most of the work in the literature is based on face recognition in controlled environments i.e., high-resolution face images with slight variation in pose, illumination and expression. CNN is used as backbone architecture in the face recognition models. Low-resolution face images having large variations in pose, illumination and expressions make a face recognition problem more challenging. It is difficult to extract discriminative features from low-resolution images. That's why the performance of face recognition models trained on high-resolution images deteriorate on low-resolution images. In this work, quality of low-resolution images is enhanced through multiple super-resolution GANs in order to improve the identification accuracy. Experiments are performed on SCface dataset and the improvement in the identification accuracy shows the effectiveness of using enhanced images generated through super-resolution GANs.

In this paper, we demonstrate simple algorithms that project low resolution (LR) images differing in subpixel shifts on a high resolution (HR) also called super resolution (SR) grid. The algorithms are very effective in accuracy as well as time efficiency. A number of spatial interpolation techniques using nearest neighbor, inverse-distance weighted averages, Radial Basis Functions (RBF) etc. are used in projection. For best accuracy of reconstructing SR image by a factor of two requires four LR images differing in four independent subpixel shifts. The algorithm has two steps: i) registration of low resolution images and (ii) shifting the low resolution images to align with reference image and projecting them on high resolution grid based on the shifts of each low resolution image using different interpolation techniques. Experiments are conducted by simulating low resolution images by subpixel shifts and subsampling of original high resolution image and the reconstructing the high resolution images from the simulated low resolution images. The results of accuracy of reconstruction are compared by using mean squared error measure between original high resolution image and reconstructed image. The algorithm was tested on remote sensing images and found to outperform previously proposed techniques such as Iterative Back Projection algorithm (IBP), Maximum Likelihood (ML) algorithms. The algorithms are robust and are not overly sensitive to the registration inaccuracies.

A new multi-frame image super resolution (SR) algorithm via Bayesian modeling with natural image prior modeled by fields of experts (FoE) is proposed. Multi-frame SR can be used to obtain a high resolution (HR) image from a set of degraded low resolution (LR) images without changing any hardware device. However, SR is well known to be an ill-posed problem. So state-of-the-art solutions usually formulate the problem with Bayesian modeling techniques, which infer the HR image based on not only the LR input images but also on prior information about the HR image. Current Bayesian SR approaches typically use simple prior models such as L1 norm, TV prior and Laplacian prior, which cannot exploit the statistics of natural scenes well. In this paper, a Bayesian multi-frame image SR approach using a FOE model as the prior for natural images is presented. The Maximum a Posteriori (MAP) framework is used for estimating the HR image. The proposed method cannot only capture the statistics of natural images well, but also require less computational power than the other Bayesian modelling methods such as Sampling methods and Approximate inference. The proposed method shows superior or comparable results to the state-of-art multi-frame SR methods.

This paper proposes a new method of synthesizing a highresolution facial image from a low-resolution facial image based on top-down learning. A face is represented by a linear combination of prototypes of shape and texture. With the shape and texture information about the pixels in a given low-resolution facial image, we can estimate optimal coefficients for a linear combination of prototypes of shape and those of texture by solving least square minimization. Then high-resolution facial image can be synthesized by using the optimal coeffcients for linear combination of the high-resolution prototypes.The encouraging results of the proposed method show that our method can be used to increase the performance of the face recognition by applying our method to enhance the low-resolution facial images captured at surveillance systems.

This work proposes a new method of enhancing the resolution of facial image from a low-resolution facial image using a recursive error back-projection of example-based learning. A face is represented by a linear combination of prototypes of shape and texture. With the shape and texture information about the pixels in a given low-resolution facial image, we can estimate optimal coefficients for a linear combination of prototypes of shape and those of texture by solving least square minimization. Then high-resolution facial image can be obtained by using the optimal coefficients for linear combination of the high-resolution prototypes, in addition to, a recursive error back-projection is applied to improve the accuracy of resolution enhancement. The encouraging results of the proposed method show that our method can be used to improve the performance of the face recognition by applying our method to enhance the low-resolution facial images captured at visual surveillance systems.

With the recent increase in intelligent CCTVs for visual surveillance, a new image degradation that integrates resolution conversion and synthetic rain models is required. For example, in heavy rain, face images captured by CCTV from a distance have significant deterioration in both visibility and resolution. Unlike traditional image degradation models (IDM), such as rain removal and super resolution, this study addresses a new IDM referred to as a scale-aware heavy rain model and proposes a method for restoring high-resolution face images (HR-FIs) from low-resolution heavy rain face images (LRHR-FI). To this end, a two-stage network is presented. The first stage generates low-resolution face images (LR-FIs), from which heavy rain has been removed from the LRHR-FIs to improve visibility. To realize this, an interpretable IDM-based network is constructed to predict physical parameters, such as rain streaks, transmission maps, and atmospheric light. In addition, the image reconstruction loss is evaluated to enhance the estimates of the physical parameters. For the second stage, which aims to reconstruct the HR-FIs from the LR-FIs outputted in the first stage, facial component-guided adversarial learning (FCGAL) is applied to boost facial structure expressions. To focus on informative facial features and reinforce the authenticity of facial components, such as the eyes and nose, a face parsing-guided generator and facial local discriminators are designed for FCGAL. The experimental results verify that the proposed approach based on a physical-based network design and FCGAL can remove heavy rain and increase the resolution and visibility simultaneously. Moreover, the proposed heavy rain face image restoration outperforms state-of-the-art models of heavy rain removal, image-to-image translation, and super resolution.

This paper presents a technique that used to enhance the image resolution. The input low resolution (LR) image is decomposed into low frequency (LF) and high frequency (HF) sub-bands by using lifting wavelet transform (LWT) and discrete wavelet transform (DWT). The resolution enhancement is done by interpolating HF sub-bands that are generated using LWT from the LR image. The HF sub-bands of DWT are used to preserve the edges of the images. The operation is done on HF sub-bands of LWT and DWT. The HF sub-bands that are generated from the DWT are interpolated and perform addition to the respective interpolated HF sub-bands of LWT. The HF sub-bands that are modified and the LR image will again interpolate and undergoes the inverse lifting wavelet transform (ILWT) to reconstruct the high resolution (HR) image. This implemented method results efficient over conventional methods.

A novel kernel-based framework for facial-image hallucination

In video surveillance, the attention human faces are frequently of small size. Image hallucination is an imperative factor disturbing the face classification by human and computer. In this paper, we propose a two-step face hallucination framework by means of training data sets which have a small quantity of low and high resolution images. In the first step, the global face is constructed based on optimal weights of training images. In the second step, a local residual compensation method bases on position patch via residual training face image data sets. Moreover, the hallucinated highresolution residual image which is obtained by the identical process can be subsequent for the local face. Finally, the hallucinated high-resolution residual image is appended with the input low-resolution face image which is interpolated to the high-resolution image dimension by an upsampling factor. Experiments fully demonstrate that our framework is very flexible and accomplishs good performance via small training data sets.

Combined variations containing low-resolution and occlusion often present in face images in the wild, e.g., under the scenario of video surveillance. While most of the existing face image recovery approaches can handle only one type of variation per model, in this work, we propose a deep generative adversarial network (FCSR-GAN) for performing joint face completion and face super-resolution via multi-task learning. The generator of FCSR-GAN aims to recover a high-resolution face image without occlusion given an input low-resolution face image with occlusion. The discriminator of FCSR-GAN uses a set of carefully designed losses (an adversarial loss, a perceptual loss, a pixel loss, a smooth loss, a style loss, and a face prior loss) to assure the high quality of the recovered high-resolution face images without occlusion. The whole network of FCSR-GAN can be trained end-to-end using our two-stage training strategy. Experimental results on the public-domain CelebA and Helen databases show that the proposed approach outperforms the state-of-the-art methods in jointly performing face super-resolution (up to 8×) and face completion, and shows good generalization ability in cross-database testing. Our FCSR-GAN is also useful for improving face identification performance when there are low-resolution and occlusion in face images. The code of FCSR-GAN is available at: https://github.com/swordcheng/FCSR-GAN.