Evolutionary latent space search for driving human portrait generation

Many GAN inversion methods have emerged to embed a given real image into the latent space of GAN for real image editing. These methods usually use a latent space composed of a series of one-dimensional vectors as an optimization space to reconstruct real images such as W+ latent space. However, the reconstructed image of these methods is usually difficult to maintain the rich detailed information in the real image. How to better preserve details in the real image is still a challenge. To solve this problem, we propose a spatially-adaptive latent space, called SA latent space, and adopt it as the optimization latent space in GAN inversion task. In particular, we use the affine transformation parameters of each convolutional layer in the generator to form the SA latent space and change affine transformation parameters from a one-dimensional vector to a spatially-adaptive three-dimensional tensor. With the more expressive latent space, we can better reconstruct the details of the real image. Extensive experiments suggest that the image reconstruction quality can be significantly improved while maintaining the semantic disentanglement ability of latent code. The code is available at https://github.com/zhang-lingyun/SalS-GAN.

In biometrics, face recognition methods are achieving momentum with recent progress in the computer vision(CV). Face recognition is widely used in the identification of an individual's identity. Unfortunately, in recent research work has revealed this face biometrics system is unprotected to spoofing attacks using by very low price instrument such as printed 2D photos attack, 3D masking attack and taking videos using smart devices (reply attack). Therefore, a Liveness Attack Detection (LAD) approach is needed to improve the high-quality security of face recognition system. Most of the earlier worked LAD methods for face anti-spoofing methods have highlight on using the handcrafted features, which are developed by expert knowledge of researcher. As example Gabor filter, Histogram of Oriented Gradients, local ternary pattern, and the Local Binary Pattern. Because of that, the extracted features consider limited factors of the problem, yielding a capture accuracy that is very low and changes with the point of presentation in attack face images. The deep learning method has developed in the computer vision research community, which is proven to be suitable for automatically training. In this article, we approach to mix or combine the handcrafted features and deep neural network features to design the discriminant face spoofing detection. The handcrafted features were based on LBP analysis. We examine the features information from the brightness and the chrominance channels using LBP descriptor. In deep features, we present an approach based on pre-trained convolutional neural network VGG-16 model using static features to recognize video and printed(2D) photo attacks. By attaching this two types of image features on our dataset and public databases, we get good results to identify real and attack images feature, called hybrid features, which has better discrimination ability to understand spoofing image feature.

Deep learning has become the main way of the object detection task for autonomous vehicles. Meanwhile, this method typically requires vast amounts of training data to reach their full potential. However, collecting the data from real world and labeling manually is an expensive, time-consuming and error-prone process. Synthetic image has the potential to replace real image for training neural networks, because image creation and labeling annotations are free in this way. For the network trained by synthetic images, the reality gap between real and synthetic images is the main obstacle to use it in the real world. And most previous works are only devoted to generate synthetic images with a good performance on model training, but lack of analysis of the domain gap that affects the performance. This work designs a method of generating the real and synthetic images to analyze and reduce the reality gap between synthetic and real images for car detection. Firstly, this work put one single car with no-background in a random background image to generate real single car and synthetic single car images. In order to further reduce the domain gap in content level, this method keeps the car distribution in synthetic images is similar with the distribution of car in real world. For the purpose of reducing the domain gap in appearance level, the parameters of camera model are same as the camera parameters of image collecting cars and the image is rendered by using the PBRT(Physically Based Ray Tracing) when we generated the synthetic images. Secondly, by training the neural network of instance segmentation with different datasets, the across validation result proves that the reality gap between synthetic and real images is no more than the domain gap between real images. Thirdly, the training results of datasets with different samples diversity show that the diversity of the samples yields better generalization between different datasets for car detection which can effectively reduce the domain gap.

Generation of photo-realistic images, semantic editing and representation learning are only a few of many applications of high-resolution generative models. Recent progress in GANs have established them as an excellent choice for such tasks. However, since they do not provide an inference model, downstream tasks such as classification cannot be easily applied on real images using the GAN latent space. Despite numerous efforts to train an inference model or design an iterative method to invert a pre-trained generator, previous methods are dataset (e.g. human face images) and architecture (e.g. StyleGAN) specific. These methods are nontrivial to extend to novel datasets or architectures. We propose a general framework that is agnostic to architecture and datasets. Our key insight is that, by training the inference and the generative model together, we allow them to adapt to each other and to converge to a better quality model. Our InvGAN, short for Invertible GAN, successfully embeds real images in the latent space of a high quality generative model. This allows us to perform image inpainting, merging, interpolation and online data augmentation. We demonstrate this with extensive qualitative and quantitative experiments.

To strengthen the security of face recognition systems to morphing attacks (MAs), many countermeasures were proposed. However, in the existing face morphing attack detection (MAD), the deep networks trained by classical score-level losses are weak in characterizing the intrinsic morphing patterns of different MAs, and they also cannot be directly applied to differential MAD scenarios. To this end, this paper presents a method for detecting and locating face MAs by the use of feature-wise supervision. It constructs the fine-grained classification loss on the basis of different morphing patterns, and designs the similarity-based and distance-based differential losses according to the properties of differential MAD scenarios. The experimental results and analysis show that the fine-grained classification loss can locate the local morphed areas after detecting MAs, while the differential losses are able to improve the generalization ability of MAD methods to unseen MAs, and can enhance the robustness of MAD methods to low-resolution and non-frontal probe face images.

Spectral alignment of multi-temporal cross-sensor images with automated kernel canonical correlation analysis

A latent space represents data by embedding them in a multidimensional vector space. In this way an abstract estimation of any complex domain could be created. Empirical approach for exploring the latent space generated by known pre-trained model of human face images using a nonlinear dimensionality reduction algorithm is presented in this paper. One aim was to find more detailed entangled features (beard and hair color) between the real images and their representation, in artistic face portrait application. Experimental results showed that sparse vectors in the latent space could be useful to obtain optimal results with relatively low effort. To evaluate our work, we present the results of a survey that was sent to 25 thousand subscribers of the real world application and got around 360 responses. The main goal of the survey was to find some quantitative measurements that can be used in our research.KeywordsLatent space explorationNonlinear dimensionality reductionGenerative adversarial networks (GANs)Industrial survey

The inversion of real images into StyleGAN's latent space is a well-studied problem. Nevertheless, applying existing approaches to real-world scenarios remains an open challenge, due to an inherent trade-off between reconstruction and editability: latent space regions which can accurately represent real images typically suffer from degraded semantic control. Recent work proposes to mitigate this trade-off by fine-tuning the generator to add the target image to well-behaved, editable regions of the latent space. While promising, this fine-tuning scheme is impractical for prevalent use as it requires a lengthy training phase for each new image. In this work, we introduce this approach into the realm of encoder-based inversion. We propose HyperStyle, a hypernetwork that learns to modulate StyleGAN's weights to faithfully express a given image in editable regions of the latent space. A naive modulation approach would require training a hypernetwork with over three billion parameters. Through careful network design, we reduce this to be in line with existing encoders. HyperStyle yields reconstructions comparable to those of optimization techniques with the near real-time inference capabilities of encoders. Lastly, we demonstrate HyperStyle's effectiveness on several applications beyond the inversion task, including the editing of out-of-domain images which were never seen during training. Code is available on our project page: https://yuval-alaluf.github.io/hyperstyle/.

Multimodal Earth observation data fusion: Graph-based approach in shared latent space

High dimensional pose state space is the main challenge in articulated human pose tracking which makes pose analysis computationally expensive or even infeasible. In this paper, we propose a novel generative approach in the framework of evolutionary computation, by which we try to widen the bottleneck with effective search strategy embedded in the extracted state subspace. Firstly, we use ISOMAP to learn the low-dimensional latent space of pose state in the aim of both reducing dimensionality and extracting the prior knowledge of human motion simultaneously. Then, we propose a manifold reconstruction method to establish smooth mappings between the latent space and original space, which enables us to perform pose analysis in the latent space. In the search strategy, we adopt a new evolutionary approach, clonal selection algorithm (CSA), for pose optimization. We design a CSA based method to estimate human pose from static image, which can be used for initialization of motion tracking. In order to make CSA suitable for motion tracking, we propose a sequential CSA (S-CSA) algorithm by incorporating the temporal continuity information into the traditional CSA. Actually, in a Bayesian inference view, the sequential CSA algorithm is in essence a multilayer importance sampling based particle filter. Our methods are demonstrated in different motion types and different image sequences. Experimental results show that our CSA based pose estimation method can achieve viewpoint invariant 3D pose reconstruction and the S-CSA based motion tracking method can achieve accurate and stable tracking of 3D human motion.

Image synthesis in magnetic resonance (MR) imaging has been an active area of research for more than ten years. MR image synthesis can be used to create images that were not acquired or replace images that are corrupted by artifacts, which can be of great benefit in automatic image analysis. Although synthetic images have been used with success in many applications, it is quite often true that they do not look like real images. In practice, an expert can usually distinguish synthetic images from real ones. Generative adversarial networks (GANs) have significantly improved the realism of synthetic images. However, we argue that further improvements can be made through the introduction of noise in the synthesis process, which better models the actual imaging process. Accordingly, we propose a novel approach that incorporates randomness into the model in order to better approximate the distribution of real MR images. Results show that the proposed method has comparable accuracy with the state-of-the-art approaches as measured by multiple similarity measurements while also being able to control the noise level in synthetic images. To further demonstrate the superiority of this model, we present results from a human observer study on synthetic images, which shows that our results capture the essential features of real MR images.

BackgroundMatrix factorization is a well established pattern discovery tool that has seen numerous applications in biomedical data analytics, such as gene expression co-clustering, patient stratification, and gene-disease association mining. Matrix factorization learns a latent data model that takes a data matrix and transforms it into a latent feature space enabling generalization, noise removal and feature discovery. However, factorization algorithms are numerically intensive, and hence there is a pressing challenge to scale current algorithms to work with large datasets. Our focus in this paper is matrix tri-factorization, a popular method that is not limited by the assumption of standard matrix factorization about data residing in one latent space. Matrix tri-factorization solves this by inferring a separate latent space for each dimension in a data matrix, and a latent mapping of interactions between the inferred spaces, making the approach particularly suitable for biomedical data mining.ResultsWe developed a block-wise approach for latent factor learning in matrix tri-factorization. The approach partitions a data matrix into disjoint submatrices that are treated independently and fed into a parallel factorization system. An appealing property of the proposed approach is its mathematical equivalence with serial matrix tri-factorization. In a study on large biomedical datasets we show that our approach scales well on multi-processor and multi-GPU architectures. On a four-GPU system we demonstrate that our approach can be more than 100-times faster than its single-processor counterpart.ConclusionsA general approach for scaling non-negative matrix tri-factorization is proposed. The approach is especially useful parallel matrix factorization implemented in a multi-GPU environment. We expect the new approach will be useful in emerging procedures for latent factor analysis, notably for data integration, where many large data matrices need to be collectively factorized.

We consider multi-solution optimization and generative models for the generation of diverse artifacts and the discovery of novel solutions. In cases where the domain's factors of variation are unknown or too complex to encode manually, generative models can provide a learned latent space to approximate these factors. When used as a search space, however, the range and diversity of possible outputs are limited to the expressivity and generative capabilities of the learned model. We compare the output diversity of a quality diversity evolutionary search performed in two different search spaces: 1) a predefined parameterized space and 2) the latent space of a variational autoencoder model. We find that the search on an explicit parametric encoding creates more diverse artifact sets than searching the latent space. A learned model is better at interpolating between known data points than at extrapolating or expanding towards unseen examples. We recommend using a generative model's latent space primarily to measure similarity between artifacts rather than for search and generation. Whenever a parametric encoding is obtainable, it should be preferred over a learned representation as it produces a higher diversity of solutions.

We consider the problem of object segmentation in cultural sites. Since collecting and labeling large datasets of real images is challenging, we investigate whether the use of synthetic images can be useful to achieve good segmentation performance on real data. To perform the study, we collected a new dataset comprising both real and synthetic images of 24 artworks in a cultural site. The synthetic images have been automatically generated from the 3D model of the considered cultural site using a tool developed for that purpose. Real and synthetic images have been labeled for the task of semantic segmentation of artworks. We compare three different approaches to perform object segmentation exploiting real and synthetic data. The experimental results point out that the use of synthetic data helps to improve the performances of segmentation algorithms when tested on real images. Satisfactory performance is achieved exploiting semantic segmentation together with image-to-image translation and including a small amount of real data during training. To encourage research on the topic, we publicly release the proposed dataset at the following url: https://iplab.dmi.unict.it/EGO-CH-OBJ-SEG/

Chest radiography is one of the most ubiquitous imaging modalities, playing an essential role in screening, diagnosis and disease management. However, chest radiography interpretation is a time-consuming and complex task, requiring the availability of experienced radiologists. As such, automated diagnosis systems for pathology detection have been proposed aiming to reduce the burden on radiologists and reduce variability in image interpretation. While promising results have been obtained, particularly since the advent of deep learning, there are significant limitations in the developed solutions, namely the lack of representative data for less frequent pathologies and the learning of biases from the training data, such as patient position, medical devices and other markers as proxies for certain pathologies. The lack of explainability is also a challenge for the adoption of these solutions in clinical practice.Generative adversarial networks could play a significant role as a solution for these challenges as they allow to artificially create new realistic images. This way, new synthetic chest radiography images could be used to increase the prevalence of less represented pathology classes and decrease model biases as well as improving the explainability of automatic decisions by generating samples that serve as examples or counter-examples to the image being analysed, ensuring patient privacy.In this study, a few-shot generative adversarial network is used to generate synthetic chest radiography images. A minimum Fréchet Inception Distance score of 17.83 was obtained, allowing to generate convincing synthetic images. Perceptual validation was then performed by asking multiple readers to classify a mixed set of synthetic and real images. An average accuracy of 83.5% was obtained but a strong dependency on reader experience level was observed. While synthetic images showed structural irregularities, the overall image sharpness was a major factor in the decision of readers. The synthetic images were then validated using a MobileNet abnormality classifier and it was shown that over 99% of images were classified correctly, indicating that the generated images were correctly interpreted by the classifier. Finally, the use of the synthetic images during training of a YOLOv5 pathology detector showed that the addition of the synthetic images led to an improvement of mean average precision of 0.05 across 14 pathologies.In conclusion, the usage of few-shot generative adversarial networks for chest radiography image generation was shown and tested in multiple scenarios, establishing a baseline for future experiments to increase the applicability of generative models in clinical scenarios of automatic CXR screening and diagnosis tools.