Face Sketch Synthesis and Recognition Based on Linear Regression Transformation and Multi-Classifier Technique

Facial expression editing in face sketch is an important and challenging problem in computer vision community as facial animation and modeling. For criminal investigation and portrait drawing, automatic expression editing tools for face sketch improve work efficiency obviously and reduce professional requirements for users. In this paper, we propose a novel method for facial expression editing in face sketch using shape space theory. The new facial expressions in the sketch images can be regenerated automatically. The method includes two components: 1) face sketch modeling; 2) expression editing. The face sketch modeling constructs 3D face sketch data from 3D facial database to match the 2D face sketch. Using facial landmarks, the "shape" of the face sketch is represented in shape space. The shape space is a manifold space which removes the rigid transform group. In shape space, the accurate 3D face sketch model is obtained which is consistent to the original 2D face sketch. For expression editing, we change the parameters of 3D face sketch model in the shape space to obtain new expressions. The expression transfer in 3D face sketch model can be mapped into the 2D face sketch. The advantages of our method are: full-automatic in modeling process; no requirements of drawing skills to user and friendly interaction; robustness to head poses and different scales. In experiments, we use the 3D facial database, FaceWareHouse, to construct the 3D face sketch model and use face sketch images from database: CUHK Face sketch Database (CUFS) to show the performance of expression editing. Experimental results demonstrate that our method can effectively edit facial expressions in face sketch with high consistency and fidelity.

This paper considers the problem of face sketch synthesis in the wild, which transforms a face photo into a face sketch. Face sketch synthesis is widely applied in law enforcement as well as digital entertainment fields. However, the existing methods either focus on hand-crafted techniques where prior human experience is relied on or adopt deep learning techniques as an end-to-end framework, where facial details cannot be well represented. In this paper, we propose a novel approach for face sketch synthesis in the wild via a deep patch representation-based probabilistic graphical model (DeepPGM). A Siamese network is constructed to extract deep patch representation from a raw facial patch, where the representative detail information for robust face sketch synthesis can be exploited. The generated deep patch representation and facial image patches are then optimally combined through a probabilistic graphical model. The proposed DeepPGM approach not only outperforms the state-of-the-art on public face sketch datasets but also can cope with forensic photos in the wild conditions, including varying lightings, poses, occlusions, skin colors, and ethnic origins. The superiority of the proposed method is demonstrated by extensive experiments on two public face sketch datasets and real-world forensic photos in the wild.

Face sketch synthesis is a crucial issue in digital entertainment and law enforcement. It can bridge the considerable texture discrepancy between face photos and sketches. Most of the current face sketch synthesis approaches directly to learn the relationship between the photos and sketches, and it is very difficult for them to generate the individual specific features, which we call rare characteristics. In this paper, we propose a novel face sketch synthesis approach through residual learning. In contrast to traditional approaches, which aim to reconstruct a sketch image directly (i.e., learn the mapping relationship between the photo and sketch), we aim to predict the residual image by learning the mapping relationship between the photo and residual, i.e., the difference between the photo and sketch, given an observed photo. This technique will render optimizing the residual mapping easier than optimizing the original mapping and deriving rare characteristic information. We also introduce a joint dictionary learning algorithm by preserving the local geometry structure of a data space. Through the learned joint dictionary, we transform the face sketch synthesis from an image space to a new and compact space; the new and compact space is spanned by learned dictionary atoms, where the manifold assumption can be further guaranteed. Results show that the proposed method demonstrates an impressive performance in the face sketch synthesis task on three public face sketch datasets and various real-world photos. These results are derived by comparing the proposed method with several state-of-the-art techniques, including certain recently proposed deep learning-based approaches.

Face sketch synthesis is a crucial technique in digital entertainment. However, the existing face sketch synthesis approaches usually generate face sketches with coarse structures. The fine details on some facial components fail to be generated. In this paper, inspired by the artists during drawing face sketches, we propose a bionic face sketch generator. It includes three parts: 1) a coarse part; 2) a fine part; and 3) a finer part. The coarse part builds the facial structure of a sketch by a generative adversarial network in the U-Net. In the middle part, the noise produced by the coarse part is erased and the fine details on the important face components are generated via a probabilistic graphic model. To compensate for the fine sketch with distinctive edge and area of shadows and lights, we learn a mapping relationship at the high-frequency band by a convolutional neural network in the finer part. The experimental results show that the proposed bionic face sketch generator can synthesize the face sketch with more delicate and striking details, satisfy the requirement of users in the digital entertainment, and provide the students with the coarse, fine, and finer face sketch copies when learning sketches. Compared with the state-of-the-art methods, the proposed approach achieves better results in both visual effects and quantitative metrics.

Face sketch synthesis, as a key technique for solving face sketch recognition, has made considerable progress in recent years. Due to the difference of modality between face photo and face sketch, traditional exemplar-based methods often lead to missed texture details and deformation while synthesizing sketches. And limited to the local receptive field, Convolutional Neural Networks-based methods cannot deal with the interdependence between features well, which makes the constraint of facial features insufficient; as such, it cannot retain some details in the synthetic image. Moreover, the deeper the network layer is, the more obvious the problems of gradient disappearance and explosion will be, which will lead to instability in the training process. Therefore, in this paper, we propose a multi-scale gradients self-attention residual learning framework for face photo-sketch transformation that embeds a self-attention mechanism in the residual block, making full use of the relationship between features to selectively enhance the characteristics of specific information through self-attention distribution. Simultaneously, residual learning can keep the characteristics of the original features from being destroyed. In addition, the problem of instability in GAN training is alleviated by allowing discriminator to become a function of multi-scale outputs of the generator in the training process. Based on cycle framework, the matching between the target domain image and the source domain image can be constrained while the mapping relationship between the two domains is established so that the tasks of face photo-to-sketch synthesis (FP2S) and face sketch-to-photo synthesis (FS2P) can be achieved simultaneously. Both Image Quality Assessment (IQA) and experiments related to face recognition show that our method can achieve state-of-the-art performance on the public benchmarks, whether using FP2S or FS2P.

In the law enforcement investigation, the police use sketching techniques to identify suspects from an eyewitness's memory. Many automatic face sketch recognition systems that determine the perpetrator’s appearance from the face image datasets have been proposed. The aim is to conduct the arrest of the right offender. We propose this work to carry out a search based on the ethnicity criterion to speed up this automatic identification and to help authorities execute fast responses by launching the retrieval process only in a part of the dataset of face images. The goal of this study is to enhance the accuracy of ethnic face sketch classification by using the convolutional neural network built on the VGG16 architecture. The FairFace dataset, which includes seven ethnic face images: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino|Hispanic, was employed in the study. We convert the face images dataset to face sketch images, and we optimize the VGG16 model for seven classification outputs. This work shows that the VGG16 deep learning model offers a reliable, automated approach for ethnic face sketch classification and recognition. The used model achieved an accuracy reaching above 94% and produced a low false negative rate, which is crucial for minimizing undetected cases.

Face sketch synthesis is a technique aimed at converting face photos into sketches. Existing face sketch synthesis research mainly relies on training with numerous photo–sketch sample pairs from existing datasets. However, these large-scale discriminative learning methods will have to face problems, such as data scarcity and high human labor costs. Once the training data become scarce, their generative performance significantly degrades. In this article, we propose a one-shot face sketch synthesis method based on diffusion models. We optimize text instructions on a diffusion model using face photo–sketch image pairs. Then, the instructions derived through gradient-based optimization are used for inference. To simulate real-world scenarios more accurately and evaluate method effectiveness more comprehensively, we introduce a new benchmark named One-shot Face Sketch Dataset (OS-Sketch). The benchmark consists of 400 pairs of face photo–sketch images, including sketches with different styles and photos with different backgrounds, ages, sexes, expressions, illumination, and so on. For a solid out-of-distribution evaluation, we select only one pair of images for training at each time, with the rest used for inference. Extensive experiments demonstrate that the proposed method can convert various photos into realistic and highly consistent sketches in a one-shot context. Compared to other methods, our approach offers greater convenience and broader applicability. The dataset will be available at: https://github.com/HanWu3125/OS-Sketch .

Face Sketch recognition has been one of the most studied topics in Forensic literature. The automatic retrieval of suspect photos from the mug-shot police data-base can help them, quickly reduce and deduct potential suspects, but in most cases, the photographic image of a suspect is not available. The best substitute is often sketch based on the memory of an eyewitness or a victim. Generally, this process is slow and really not effective, it does not allow to find and arrest the right suspect. So, a stronger algorithm for even partial face sketch recognition can be useful. Although many methods have been proposed in this scenario, especially the techniques that are applied to face recognition system and ranked among the perfect and most effective. The main objective of this paper is to presents a review of recent and different researches on recognizing face sketches. We wrote this work to discover and analyze multiple researches on face sketch recognition. We wrote it to review the different classes of approaches, of datasets and evaluation protocols used, by analyzing their kernels and identifying their limitations.

This research provides a strategy for enhancing the precision of face sketch identification through adversarial sketch-photo transformation. The approach uses a generative adversarial network (GAN) to learn to convert sketches into photographs, which may subsequently be utilized to enhance the precision of face sketch identification. The suggested method is evaluated in comparison to state-of-the-art face sketch recognition and synthesis techniques, such as sketchy GAN, similarity-preserving GAN (SPGAN), and super-resolution GAN (SRGAN). Possible domains of use for the proposed adversarial sketch-photo transformation approach include law enforcement, where reliable face sketch recognition is essential for the identification of suspects. The suggested approach can be generalized to various contexts, such as the creation of creative photographs from drawings or the conversion of pictures between modalities. The suggested method outperforms state-of-the-art face sketch recognition and synthesis techniques, confirming the usefulness of adversarial learning in this context. Our method is highly efficient for photo-sketch synthesis, with a structural similarity index (SSIM) of 0.65 on The Chinese University of Hong Kong dataset and 0.70 on the custom-generated dataset.

Face sketch synthesis has many practical applications, such as law enforcement and digital entertainment. Existing face sketch synthesis methods focus on neighbor selection and/or weight reconstruction. However, these approaches did not take “interpretation through synthesis” into consideration obviously. Active appearance model (AAM) is one of “interpretation through synthesis” approaches. In this paper, we introduce AAM to “explain” face photos by generating synthetic images that are as similar as possible. Then AAM provides a compact set of parameters that are useful for face sketch synthesis. Extensive experiments on public face sketch databases demonstrate the superiority of the proposed method in comparison to state-of-the-art methods.

This paper presents a simple Face Sketch-Photo Synthesis and Recognition system. Face Sketch Synthesis provides a way to compare and match the faces present in two different modalities (i.e. face-sketches and face-photos). The aim is to significantly reduce the differences between face-sketches and face-photos and also decrease the texture irregularity between them by converting photos to sketches and vice-versa. This results in effective matching between the two thus simplifying the process of facial recognition. This system is modeled using three major components: (i) For a given input face-photo, obtaining an output face-sketch. It is designed using image processing techniques like 2 scale image decomposition and color dodging. (ii) For a given input face-sketch, obtaining an output face-photo. Convolutional Neural Networks are used to model this component. (iii) For a given query face-sketch or face-photo, recognition of face-photo or face-sketch in the database. It is implemented using Fisherface Linear Discriminant Analysis.KeywordsFace-sketch synthesisFace-photo synthesisFace-sketch recognitionFace-photo recognitionConvolutional Neural NetworksFisherface Linear Discriminant Analysis

Face recognition systems have grown in popularity worldwide due to technological advancements. The use of a sketch to identify a suspect is one of the most common approaches in the field of forensic science when it comes to the application of face recognition algorithms. The difficulties arise when no image capture sources are available. Without an image, it becomes challenging for the police to identify the suspect. In forensic sciences, the identification of face sketch is used when the eyewitness of the crime scene is asked to recall the structures and features of the suspected person, and then the forensic artist will generate a sketch based on the verbal description provided from the eyewitness. Matching these sketches to the corresponding photo is a difficult task. Even though the sketches contain all of the necessary information about the suspect face's spatial topology and geometric details, it is still difficult for manual face recognition to identify and match sketch representations to the corresponding photo. Therefore, an automatic matching method is needed to identify a suspect accurately. This study perform a systematic literature review (SLR) that focuses on discussing of three major aspects of the face sketch recognition system which are the available face sketch database, the recent technique used sketch-photo matching and the contribution of the Generative Adversarial Network (GAN) in face sketch recognition. In this study, 35 different research papers published between 2011 and 2021 are studied and analysed. From this SLR, compared to other methods, GAN had been found to be a great technique for solving a wide variety of general facial recognition problems and also face synthesis.

In this paper, we propose a novel face photo-sketch synthesis and recognition method using a multiscale Markov Random Fields (MRF) model. Our system has three components: 1) given a face photo, synthesizing a sketch drawing; 2) given a face sketch drawing, synthesizing a photo; and 3) searching for face photos in the database based on a query sketch drawn by an artist. It has useful applications for both digital entertainment and law enforcement. We assume that faces to be studied are in a frontal pose, with normal lighting and neutral expression, and have no occlusions. To synthesize sketch/photo images, the face region is divided into overlapping patches for learning. The size of the patches decides the scale of local face structures to be learned. From a training set which contains photo-sketch pairs, the joint photo-sketch model is learned at multiple scales using a multiscale MRF model. By transforming a face photo to a sketch (or transforming a sketch to a photo), the difference between photos and sketches is significantly reduced, thus allowing effective matching between the two in face sketch recognition. After the photo-sketch transformation, in principle, most of the proposed face photo recognition approaches can be applied to face sketch recognition in a straightforward way. Extensive experiments are conducted on a face sketch database including 606 faces, which can be downloaded from our Web site (http://mmlab.ie.cuhk.edu.hk/facesketch.html).

Face sketch synthesis has made significant progress with the development of deep neural networks in these years. The delicate depiction of sketch portraits facilitates a wide range of applications like digital entertainment and law enforcement. However, accurate and realistic face sketch generation is still a challenging task due to the illumination variations and complex backgrounds in the real scenes. To tackle these challenges, we propose a novel Semantic-Driven Generative Adversarial Network (SDGAN) which embeds global structure-level style injection and local class-level knowledge re-weighting. Specifically, we conduct facial saliency detection on the input face photos to provide overall facial texture structure, which could be used as a global type of prior information. In addition, we exploit face parsing layouts as the semantic-level spatial prior to enforce globally structural style injection in the generator of SDGAN. Furthermore, to enhance the realistic effect of the details, we propose a novel Adaptive Re-weighting Loss (ARLoss) which dedicates to balance the contributions of different semantic classes. Experimentally, our extensive experiments on CUFS and CUFSF datasets show that our proposed algorithm achieves state-of-the-art performance.

Presently, suspect prediction of crime scenes can be considered as a classification task, which predicts the suspects based on the time, space, and type of crime. Performing digital forensic investigation in a big data environment poses several challenges to the investigational officer. Besides, the facial sketches are widely employed by the law enforcement agencies for assisting the suspect identification of suspects involved in crime scenes. The sketches utilized in the forensic investigations are either drawn by forensic artists or generated through the computer program (composite sketches) based on the verbal explanation given by the eyewitness or victim. Since this suspect identification process is slow and difficult, it is required to design a technique for a quick and automated facial sketch generation. Machine Learning (ML) and deep learning (DL) models find it useful to automatically support the decision of forensics experts. The challenge is the incorporation of the domain expert knowledge with DL models for developing efficient techniques to make better decisions. In this view, this study develops a new artificial intelligence (AI) based DL model with face sketch synthesis (FSS) for suspect identification (DLFSS-SI) in a big data environment. The proposed method performs preprocessing at the primary stage to improvise the image quality. In addition, the proposed model uses a DL based MobileNet (MN) model for feature extractor, and the hyper parameters of the MobileNet are tuned by quasi oppositional firefly optimization (QOFFO) algorithm. The proposed model automatically draws the sketches of the input facial images. Moreover, a qualitative similarity assessment takes place with the sketch drawn by a professional artist by the eyewitness. If there is a higher resemblance between the two sketches, the suspect will be determined. To validate the effective performance of the DLFSS-SI method, a detailed qualitative and quantitative examination takes place. The experimental outcome stated that the DLFSS-SI model has outperformed the compared methods in terms of mean square error (MSE), peak signal to noise ratio (PSNR), average actuary, and average computation time.