Task-Aware Compressed Sensing With Generative Adversarial Networks

Thanks to the availability of large-scale data, deep Convolutional Neural Networks (CNNs) have witnessed success in various applications of computer vision. However, the performance of CNNs on Synthetic Aperture Radar (SAR) image classification is unsatisfactory due to the lack of well-labeled SAR data, as well as the differences in imaging mechanisms between SAR images and optical images. Therefore, this paper addresses the problem of SAR image classification by employing the Generative Adversarial Network (GAN) to produce more labeled SAR data. We propose special GANs for generating SAR images to be used in the training process. First, we incorporate the quadratic operation into the GAN, extending the convolution to make the discriminator better represent the SAR data; second, the statistical characteristics of SAR images are integrated into the GAN to make its value function more reasonable; finally, two types of parallel connected GANs are designed, one of which we call PWGAN, combining the Deep Convolutional GAN (DCGAN) and Wasserstein GAN with Gradient Penalty (WGAN-GP) together in the structure, and the other, which we call CNN-PGAN, applying a pre-trained CNN as a discriminator to the parallel GAN. Both PWGAN and CNN-PGAN consist of a number of discriminators and generators according to the number of target categories. Experimental results on the TerraSAR-X single polarization dataset demonstrate the effectiveness of the proposed method.

Computer vision is one of the hottest research fields in deep learning. The emergence of generative adversarial networks (GANs) provides a new method and model for computer vision. The idea of GANs using the game training method is superior to traditional machine learning algorithms in terms of feature learning and image generation. GANs are widely used not only in image generation and style transfer but also in the text, voice, video processing, and other fields. However, there are still some problems with GANs, such as model collapse and uncontrollable training. This paper deeply reviews the theoretical basis of GANs and surveys some recently developed GAN models, in comparison with traditional GAN models. The applications of GANs in computer vision include data enhancement, domain transfer, high-quality sample generation, and image restoration. The latest research progress of GANs in artificial intelligence (AI) based security attack and defense is introduced. The future development of GANs in computer vision is also discussed at the end of the paper with possible applications of AI in computer vision.

Magnetic Resonance Imaging (MRI) is one of the most commonly used modalities in medical imaging, and one of the main limitations is its slow scanning and reconstruction speed. Recently, deep learning used for compressed sensing (CS) methods have been proposed to accelerate the acquisition by undersampling in the K-space and reconstruct images with neural networks. However, there are still some challenges remained: First, directly training networks based on L1/L2 distance to the target fully sampled images may lead to fuzzy reconstruction images because L1/L2 loss only enforces the overall image or patch similarity, but does not consider the local details such as anatomical sharpness. Second, Generative Adversarial Networks (GAN) can partially solve this problem. The undersampling image gets the latent space through the encoder, and the image is reconstructed by the decoder based on GAN loss, but it may generate unrealistic details by lacking of constraints in K-space domain. Third, most of the networks after training are fixed and have limited adaptation capability in the inference time, and the patient-specific information cannot be effectively used. To resolve these challenges, we proposed a new compressed sensing GAN reconstruction method, and there are two main contributions: (1) We proposed a encoder-decoder structure, which guided GAN optimization strategy data-consistency in latent space to improve the reconstruction quality such as preserving more local details and improving the anatomical sharpness while constraining GAN to follow the data distribution in the K-space to prevent the unrealistic details. (2) An online update strategy is used to find the best representation in the latent space for the underlying patient, and the reconstruct result can be further improved by incorporating the patient-specific information. Extensive experimental results show the effectiveness of our method. KeywordsMRI reconstructionData-consistencyLatent spaceOnline update

In a brief period, the rapid increase in interest in Computer Vision and Deep Learning has led to a plethora of different applications on text, images, and videos. Those applications range from simple problems such as “motion detection in static cameras” or “Spam Filtering” to more complex ones such as “Robot Object Grasping through Vision” or “Image Caption Generation.” With the expeditious development of Computer Graphics, and 3D models acquisition technologies, Deep Learning applications on 3D Object Models have attracted increased attention. Sensors can provide us with 3D data with rich geometry, shape, and scale information, and accompanied by 2D images, can grant us with a better understanding of a certain environment. Three-Dimensional Data can be represented in a variety of ways, one of the most commonly used being point clouds (PC), which preserve the primary geometric information in 3D space without any discretization. However, even Deep Learning on Point Clouds is still in its preliminary stages due to challenges encountered when processing PCs with Deep Neural Networks. One of the main challenges is that decent quality PC Models are hard to obtain and often miss parts of data. A way to solve this problem is to create a 3D Shape Completion Model capable of restoring those missing parts. In this dissertation, we will research a family of models, with a variety of applications in Computer Vision, called Deep Generative Models. Specifically, we will expand indepth on Generative Adversarial Networks (GANs) as well as Variational Autoencoders (VAE). GANs create new data based on a set of given training data, with which they have the same statistics, whereas VAEs ensure that their encodings distribution is regularized during the training so that the latent space is of sufficient quality, to generate new data. The main objective is to create a point-based shape completion network based on Deep Generative Models that can complete a partial scheme with reasonable results.

In the field of image forensics, notable attention has been recently paid toward the detection of synthetic contents created through Generative Adversarial Networks (GANs), especially face images. This work explores a classification methodology inspired by the inner architecture of typical GANs, where vectors in a low-dimensional latent space are transformed by the generator into meaningful high-dimensional images. In particular, the proposed detector exploits the inversion of the GAN synthesis process: given a face image under investigation, we identify the point in the GAN latent space which more closely reconstructs it; we project the vector back into the image space, and we compare the resulting image with the actual one. Through experimental tests on widely known datasets (including FFHQ, CelebA, LFW, and Caltech), we demonstrate that real faces can be accurately discriminated from GAN-generated ones by properly capturing the facial traits through different feature representations. In particular, features based on facial landmarks fed to a Support Vector Machine consistently yield a global accuracy of above 88% for each dataset. Furthermore, we experimentally prove that the proposed detector is robust concerning routinely applied post-processing operations.

<p>Despite continuous improvements in numerical atmospheric models and the ever-growing computational resources, global weather forecast and climate models still operate with grid spacings between 10 and 50 km. While a further increase in spatial resolution is computationally expensive and introduces new challenges in the design of these models (e.g. gray-zone parameterizations), statistical downscaling models can overcome the mentioned issues. Inspired by the success of super-resolution applications in computer vision, deep neural networks have recently been recognized as an appealing approach for statistical downscaling of meteorological fields.</p><p>In this study, we apply a generative adversarial network (GAN) to downscale the 2m temperature over Central Europe where complex terrain introduces a high degree of spatial variability. GANs are considered superior to purely convolutional networks since the model is encouraged to generate data whose statistical properties are similar to real data. Here, the generator consists of an u-shaped encoder-decoder network which is capable of extracting features on various spatial scales. Instead of pursuing a pure super-resolution approach where coarsened data is mapped back to high resolution data, we choose a more realistic, but also more challenging test suite. Coarsened ERA5 reanalysis data, originally provided on a 0.3°-grid, is used to downscale the 2m temperature onto a 0.1°-grid with the help of short-range forecasts from the operational Integrated Forecasting System (IFS) model. Thus, our GAN model does not only have to reconstruct finer spatial features, but must also correct for model biases.  <br>The downscaled 2m temperature is evaluated in terms of several evaluation metrics measuring the error on grid point-level as well as the quality in terms of spatial variability and the produced probability function. We also investigate the importance of static and dynamic predictors such as the surface elevation and the temperature on different pressure levels, respectively. Our results motivate further development of deep neural networks for statistical downscaling of meteorological fields. This includes downscaling of other, inherently uncertain variables such as precipitation, operations on spatial resolutions at kilometer-scale and ultimately targets an operational application on output data from global NWP models.</p>

<p>Inspired by the success of superresolution applications in computer vision, deep neural networks have recently been recognized as an appealing approach for statistical downscaling of meteorological fields. While further increasing the resolution of numerical weather prediction models is computationally very expensive, statistical downscaling models can accomplish this task much cheaper once they have been trained.</p><p>In this study, we apply a generative adversarial network (GAN) to downscale the 2m temperature over Central Europe where complex terrain introduces a high degree of spatial variability. GANs are considered superior to purely convolutional networks since the model is encouraged to generate data whose statistical properties are similar to real data. Here, the generator consists of an u-shaped encoder decoder network which is capable of extracting features on various spatial scales. As a quasi-realistic test suite, we map data from the ERA5 reanalysis dataset onto a 0.1°-grid with the help of short-range forecasts from the Integrated Forecasting System (IFS) model. To increase the complexity of the downscaling task, the ERA5 reanalysis data is coarsened beforehand onto a 0.8°-grid, thus increasing the downscaling factor to 8. We evaluate our statistical downscaling model in terms of several evaluation metrics which measure the error on grid point-level as well as the quality of the downscaled product in terms of spatial variability and produced probability function. We also investigate the importance of static and dynamic predictors such as the surface elevation and the temperature on different pressure levels, respectively. Our results motivate further development of deep neural networks for statistical downscaling of meteorological fields. This includes downscaling of other, inherently uncertain variables such as precipitation, operations on spatial resolutions at kilometer-scale and ultimately targets an operational application on output data from global NWP models.</p>

Generative Adversarial Network (GAN) is neural network architecture, widely used in many computer vision applications such as super-resolution image generation, art creation and image to image translation. A conventional GAN model consists of two sub-models; generative model and discriminative model. The former one generates new samples based on an unsupervised learning task, and the later one classifies them into real or fake. Though GAN is most commonly used for training generative models, it can be used for developing a classifier model. The main objective is to extend the effectiveness of GAN into semi-supervised learning, i.e., for the classification of fundus images to diagnose glaucoma. The discriminator model in the conventional GAN is improved via transfer learning to predict n + 1 classes by training the model for both supervised classification (n classes) and unsupervised classification (fake or real). Both models share all feature extraction layers and differ in the output layers. Thus any update in one of the model will impact both models. Results show that the semi-supervised GAN performs well than a standalone Convolution Neural Networks (CNNs) model.

In the field of deep learning, the generative model did not attract much attention until GANs (generative adversarial networks) appeared. In 2014, Google’s Ian Goodfellow proposed a generative model called GANs. GANs use different structures and objective functions from the existing generative model. For example, GANs use two neural networks: a generator that creates a realistic image, and a discriminator that distinguishes whether the input is real or synthetic. If there are no problems in the training process, GANs can generate images that are difficult even for experts to distinguish in terms of authenticity. Currently, GANs are the most researched subject in the field of computer vision, which deals with the technology of image style translation, synthesis, and generation, and various models have been unveiled. The issues raised are also improving one by one. In image synthesis, BEGAN (Boundary Equilibrium Generative Adversarial Network), which outperforms the previously announced GANs, learns the latent space of the image, while balancing the generator and discriminator. Nonetheless, BEGAN also has a mode collapse wherein the generator generates only a few images or a single one. Although BEGAN-CS (Boundary Equilibrium Generative Adversarial Network with Constrained Space), which was improved in terms of loss function, was introduced, it did not solve the mode collapse. The discriminator structure of BEGAN-CS is AE (AutoEncoder), which cannot create a particularly useful or structured latent space. Compression performance is not good either. In this paper, this characteristic of AE is considered to be related to the occurrence of mode collapse. Thus, we used VAE (Variational AutoEncoder), which added statistical techniques to AE. As a result of the experiment, the proposed model did not cause mode collapse but converged to a better state than BEGAN-CS.

AI and computer vision are making strides every day with newer solutions to real-world problems and emerging challenges. An extremely important and useful application of computer vision is surveillance systems for security in open or closed spaces. This research work investigates a security application by forecasting crowd trajectories using surveillance videos. Generative Adversarial Networks(GANs) are commonly used for video processing applications like future frame synthesis. Social GAN(SGAN) is one of the recent models that has been used for crowd trajectory prediction. However, SGAN is a two stream architecture that keeps the background and foreground separate and predicts the foreground changes only, keeping the background static. In this paper, a novel GAN architecture called iSGAN is proposed for crowd trajectory prediction. The proposed model is an improvement of SGAN and it incorporates the onestream architecture of improved Video GAN (iVGAN) into SGAN. The one-stream architecture doesn't separate the backgroungd and foreground and thus allowing to captiure the dynamics of the video (camera zoom, pan, lighting change, obstacles etc.). The proposed iSGAN model has been assessed on the benchmark ETH Pedestrian dataset and it is found that the iSGAN Model outperforms other existing trajectory prediction models.

Generative Adversarial Networks (GANs) has shown its dramatical success, especially in computer vision applications. In this paper, inspired by traditional GANs, we propose Multi-GANs which is an architecture of multiple generative adversarial networks that works together. Whilst, the GANs are successful to generate images which looks realistic but the real-world problems are much more complicated than a GANs can perform a desirable outcome to the whole of the problem space. Therefore, our approach divides each problem space into the several smaller and of course much more homogeneous subspaces. We propose then a GANs for each sub-space that can learn to mimic any distribution of data with lower lost. The results of each GANs for all sub-spaces then merge together to perform the original preliminary space. We evaluated our approach on Pseudo-Coloring which is a very difficult and ill-posed problem among the computer vision community. The experimental results show much more realistic characteristics for the generated images also its superiority in comparison to the traditional approaches.

Quantitative and flexible 3D shape dataset augmentation via latent space embedding and deformation learning

The mining belt conveyor is one of the most important modules in coal mine, whose safety always be threatened by the foreign objects. Although the traditional target detection methods achieve promising results in various computer vision tasks, the performance heavily depends on sufficient labelled data. However, in real-world production scenario, it is difficult to acquire huge number of images with foreign objects. The obtained datasets lacking of capacity and diversity are not suitable for training supervised learning-based foreign objects detection models. To address this concern, we propose a novel method for detecting the foreign objects on the surface of underground coal conveyor belt via improved GANomaly. The proposed foreign objects detection method employs generative adversarial networks (GAN) with attention gate to capture the distribution of normality in both high-dimensional image space and low-dimensional latent vector space. Only the normal images without foreign object are utilized to adversarially train the proposed network, including a U-shape generator to reconstruct the input image and a discriminator to classify real images from reconstructed ones. Then the combination of the difference between the input and generated images as well as the difference between latent representations are utilized as the anomaly score to evaluate whether the input image contain foreign objects. Experimental results over 707 images from real-world industrial scenarios demonstrate that the proposed method achieves an area under the receiver operating characteristic curve of 0.864 and is superior to the previous GAN-based anomaly detection methods.

Modifying facial attributes without the paired dataset proves to be a challenging task. Previously, approaches either required supervision from a ground-truth transformed image or required training a separate model for mapping every pair of attributes. These limit the scalability of the models to accommodate a larger set of attributes since the number of models that we need to train grows exponentially large. Another major drawback of the previous approaches is the unintentional gain of the identity of the person as they transform the facial attributes. We propose a method that allows for controllable and identity-aware transformations across multiple facial attributes using only a single model. Our approach is to train a generative adversarial network (GAN) with a multitask conditional discriminator that recognizes the identity of the face, distinguishes real images from fake, as well as identifies facial attributes present in an image. This guides the generator into producing an output that is realistic while preserving the person's identity and facial attributes. Through this framework, our model also learns meaningful image representations in a lower dimensional latent space and semantically associate separate parts of the encoded vector with both the person's identity and facial attributes. This opens up the possibility of generating new faces and other transformations such as making the face thinner or chubbier. Furthermore, our model only encodes the image once and allows for multiple transformations using the encoded vector. This allows for faster transformations since it does not need to reprocess the entire image for every transformation. We show the effectiveness of our proposed method through both qualitative and quantitative evaluations, such as ablative studies, visual inspection, and face verification. Competitive results are achieved compared to the main competition (CycleGAN), however, at great space and extensibility gain by using a single model.

Autoencoders and generative neural network models have recently gained popularity in fluid mechanics due to their spontaneity and low processing time instead of high fidelity CFD simulations. Auto encoders are used as model order reduction tools in applications of fluid mechanics by compressing input high-dimensional data using an encoder to map the input space into a lower-dimensional latent space. Whereas, generative models such as Variational Auto-encoders (VAEs) and Generative Adversarial Networks (GANs) are proving to be effective in generating solutions to chaotic models with high ‘randomness’ such as turbulent flows. In this study, forced isotropic turbulence flow is generated by parameterizing into some basic statistical characteristics. The models trained on pre-simulated data from dependencies on these characteristics and the flow generation is then affected by varying these parameters. The latent vectors pushed along the generator models like the decoders and generators contain independent entries which can be used to create different outputs with similar properties. The use of neural network-based architecture removes the need for dependency on the classical mesh-based Navier-Stoke equation estimation which is prominent in many CFD softwares.