Controllable Image Dataset Construction Using Conditionally Transformed Inputs in Generative Adversarial Networks

In the field of deep learning, the generative model did not attract much attention until GANs (generative adversarial networks) appeared. In 2014, Google’s Ian Goodfellow proposed a generative model called GANs. GANs use different structures and objective functions from the existing generative model. For example, GANs use two neural networks: a generator that creates a realistic image, and a discriminator that distinguishes whether the input is real or synthetic. If there are no problems in the training process, GANs can generate images that are difficult even for experts to distinguish in terms of authenticity. Currently, GANs are the most researched subject in the field of computer vision, which deals with the technology of image style translation, synthesis, and generation, and various models have been unveiled. The issues raised are also improving one by one. In image synthesis, BEGAN (Boundary Equilibrium Generative Adversarial Network), which outperforms the previously announced GANs, learns the latent space of the image, while balancing the generator and discriminator. Nonetheless, BEGAN also has a mode collapse wherein the generator generates only a few images or a single one. Although BEGAN-CS (Boundary Equilibrium Generative Adversarial Network with Constrained Space), which was improved in terms of loss function, was introduced, it did not solve the mode collapse. The discriminator structure of BEGAN-CS is AE (AutoEncoder), which cannot create a particularly useful or structured latent space. Compression performance is not good either. In this paper, this characteristic of AE is considered to be related to the occurrence of mode collapse. Thus, we used VAE (Variational AutoEncoder), which added statistical techniques to AE. As a result of the experiment, the proposed model did not cause mode collapse but converged to a better state than BEGAN-CS.

Generative Adversarial Networks (GANs) have received a great deal of attention due in part to recent success in generating original, high-quality samples from visual domains. However, most current methods only allow for users to guide this image generation process through limited interactions. In this work we develop a novel GAN framework that allows humans to be “in-the-loop” of the image generation process. Our technique iteratively accepts relative constraints of the form “Generate an image more like image A than image B”. After each constraint is given, the user is presented with new outputs from the GAN, informing the next round of feedback. This feedback is used to constrain the output of the GAN with respect to an underlying semantic space that can be designed to model a variety of different notions of similarity (e.g. classes, attributes, object relationships, color, etc.). In our experiments, we show that our GAN framework is able to generate images that are of comparable quality to equivalent unsupervised GANs while satisfying a large number of the constraints provided by users, effectively changing a GAN into one that allows users interactive control over image generation without sacrificing image quality.

The Deepfake phenomenon has become very popular nowadays thanks to the possibility to create incredibly realistic images using deep learning tools, based mainly on ad-hoc Generative Adversarial Networks (GAN). In this work we focus on the analysis of Deepfakes of human faces with the objective of creating a new detection method able to detect a forensics trace hidden in images: a sort of fingerprint left in the image generation process. The proposed technique, by means of an Expectation Maximization (EM) algorithm, extracts a set of local features specifically addressed to model the underlying convolutional generative process. Ad-hoc validation has been employed through experimental tests with naive classifiers on five different architectures (GDWCT, STARGAN, ATTGAN, STYLEGAN, STYLEGAN2) against the CELEBA dataset as ground-truth for non-fakes. Results demonstrated the effectiveness of the technique in distinguishing the different architectures and the corresponding generation process.

ECG signal generation based on conditional generative models

We consider the task of photo-realistic unconditional image generation (generate high quality, diverse samples that carry the same visual content as the image) on mobile platforms using Generative Adversarial Networks (GANs). In this paper, we propose a novel approach to trade-off image generation accuracy of a GAN for the energy consumed (compute) at run-time called Scale-Energy Trade-off GAN (SETGAN). GANs usually take a long time to train and consume a huge memory hence making it difficult to run on edge devices. The key idea behind SETGAN for an image generation task is for a given input image, we train a GAN on a remote server and use the trained model on edge devices. We use SinGAN, a single image unconditional generative model, that contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. During the training process, we determine the optimal number of scales for a given input image and the energy constraint from target edge device. Results show that with the SETGAN's unique client-server based architecture, we were able to achieve 56% gain in energy for a loss of 3% to 12% SSIM accuracy. Also, with the parallel multi-scale training, we obtain around 4x gain in training time on the server.

Generative Adversarial Networks (GANs) are a major advancement in generative modeling, surpassing traditional machine learning models in tasks such as image generation, super-resolution, and image-to-text translation. A GAN consists of two neural networks: a Generator (G), which creates data from noise or a latent vector, and a Discriminator (D), which determines whether the data is real or generated. These networks train competitively, improving each other iteratively to produce increasingly realistic outputs. However, GANs face challenges like mode collapse, unstable training, and convergence issues, leading to the adoption of strategies such as instance normalization and enhanced loss functions. Future research can focus on improving stability, developing novel loss functions, and applying GANs in unsupervised learning. Performance metrics like Inception Score, Fréchet Inception Distance (FID), and Structural Similarity Index (SSIM) are essential for evaluating and comparing GAN architectures. Additionally, ethical concerns, including the misuse of GANs for deepfakes and synthetic data, underscore the importance of transparency, accountability, and ethical standards in research and deployment. This review provides an accessible introduction to GANs for novice researchers, along with a detailed analysis of their limitations, applications, and future prospects, offering valuable insights and direction for advancing this field.

Diffusion models, variational autoencoders, and generative adversarial networks (GANs) are three common types of generative artificial intelligence models for image generation. Among these, GANs are the most frequently used for medical image generation and are often employed for data augmentation in various studies. However, due to the adversarial nature of GANs, where the generator and discriminator compete against each other, the training process can sometimes end with the model unable to generate meaningful images or even producing noise. This phenomenon is rarely discussed in the literature, and no studies have proposed solutions to address this issue. Such outcomes can introduce significant bias when GANs are used for data augmentation in medical image training. Moreover, GANs often require substantial computational power and storage, adding to the challenges. In this study, we used deep convolutional GANs for chest X-raygeneration, and three typical training outcomes were found. Two scenarios generated meaningful medical images and one failed to produce usable images. By analyzing the loss history during training, we observed that the regression line of the overall losses tends to diverge slowly. After excluding outlier losses, we found that the slope of the regression line within the stable loss segment indicates the optimal point to terminate training, ensuring the generation of meaningful medical images.

Image synthesis with adversarial networks: A comprehensive survey and case studies

As a promising generative modeling method, Generative Adversarial Networks are a deep-learning-based generative model, in which two networks, namely the generative network and the discriminant network, contest with each other in a game during which the generator “learns” and get trained to be able to fool the discriminator of believing a specific image is (superficially) authentic. It is widely used to generate pictures based on existing data. Based on the MNIST data set, this paper studies and analyzes the application of Generative Adversarial Networks in the generation of handwritten digital images. In addition, by studying the Deep Convolutional Generative Adversarial Network and the Conditional Generation Adversarial Network model, we compared the connections and differences between these three models in training the MNIST dataset to generate handwritten digits algorithms, and gave an optimization method for generating handwritten digits.

Generative Adversarial Network (GANs) is widely implemented not just for image generation but also text, audio, video processing, and more. To add on, methods of image restoration and enhancement such as semantic inpainting, outpainting, filtering, and denoising are achievable by GANs. This paper reviews the theoretical basis of GANs in image generation, also known as image synthesis, and possibilities of successful restoration of historical illustrations such as buildings, landmarks, and cultural heritage that have been lost or demolished. By training GANs on a relevant dataset of images, it can learn characteristics and features of the dataset to generate desired results based on raw image as input. Theoretical knowledge about GANs and relevancy of study is elaborated. For methodology, random sampling and stratified sampling are applied for data collection with justifications together with limitations. The proposed system, HistoGAN, restores damaged or incomplete images through image synthesis after being trained with large datasets. Performance of implementation is analyzed and evaluated to identify space of improvement to uplift performance. HistoGAN has the potential to be applicable in other fields such as historical research, architecture discovery, and education.

Synthesis of Image from Text using Generative Adversarial Networks

This study identifies the principles of image generation artificial intelligence, examines its performance and limitations, and examines the innovative changes that have occurred in image production methods. Specifically, we reveal the structure of the Generative Adversarial Network (GAN), trace its technical improvement process in detail, and classify the types of subsequent models developed later. The tasks performed by GAN’s successor models are diverse. GAN is steadily evolving, completing amazing tasks that were impossible with existing generative models, such as high-resolution image generation, image substitution such as drawings and photos, and face synthesis. The principles of GAN are, first, data learning through adversarial competition between generator and discriminator, second, calculation of accurate probability distribution through Maximum Likelihood Estimation, and third, calculation of fine features of images through deep convolution structure. The most important characteristic of GAN as a generative model is its ability to create realistic images similar to photographs. Humans have been producing images using hands and tools, but with the invention of photography, humans began producing images using camera devices. Now, generative artificial intelligence is becoming another means of image production as technology continues to improve rapidly.

Generative AI models have emerged as transformative tools in the healthcare domain, offering significant advances in disease detection, diagnosis, prognosis, and treatment planning. Although substantial progress has been made in the generation of 2D medical images, the synthesis of complex medical videos remains an unexplored area. The available literature on the generation of synthetic medical videos is minimal, highlighting a significant gap in this emerging area of research. This paper reviews the literature related to biomedical video synthesis using diffusion models and generative adversarial networks. The review aims to consolidate all relevant literature and highlight the different publicly available datasets, performance matrices, and the challenges associated with the generation of medical videos, along with some potential mitigation strategies. The findings of this review reveal that key challenges, such as maintaining temporal consistency, addressing computational inefficiencies, and overcoming data scarcity, are interconnected issues. Addressing these issues collectively is essential for the development of accurate and robust generative models tailored for medical video synthesis. The proposed potential mitigation strategies for the limitations of generative models in this review serve as a foundational resource for future research, aiming to enhance the reliability and applicability of generative AI models in clinical settings. These advances have the potential to significantly impact the domains of connected healthcare and personalized medicine by enabling the generation of realistic, high-quality medical video data that can enhance the training of diagnostic algorithms, improve the robustness of AI-assisted video interpretation, simulate disease progression or regression for more precise treatment planning, and support the development of personalized medicine techniques through enriched longitudinal data analysis.

The Generative Adversarial Network (GAN) is a state-of-the-art technique in the field of deep learning. A number of recent papers address the theory and applications of GANs in various fields of image processing. Fewer studies, however, have directly evaluated GAN outputs. Those that have been conducted focused on using classification performance, e.g., Inception Score (IS) and statistical metrics, e.g., Fr\'echet Inception Distance (FID). Here, we consider a fundamental way to evaluate GANs by directly analyzing the images they generate, instead of using them as inputs to other classifiers. We characterize the performance of a GAN as an image generator according to three aspects: 1) Creativity: non-duplication of the real images. 2) Inheritance: generated images should have the same style, which retains key features of the real images. 3) Diversity: generated images are different from each other. A GAN should not generate a few different images repeatedly. Based on the three aspects of ideal GANs, we have designed the Likeness Score (LS) to evaluate GAN performance, and have applied it to evaluate several typical GANs. We compared our proposed measure with two commonly used GAN evaluation methods: IS and FID, and four additional measures. Furthermore, we discuss how these evaluations could help us deepen our understanding of GANs and improve their performance.

Deep learning methods have made remarkable strides in surface defect detection. But, they heavily rely on large amount of training data, which can be a costly endeavor, especially for specific applications like steel strip surface defect detection, where acquiring and labeling large-scale data is impractical due to the rarity of certain defective categories in production environment. Hence, realistic defect image synthesis can greatly alleviate this issue. However, training image generation networks also demand substantial data, making image data augmentation merely an auxiliary effort. In this work, we propose a Generative Adversarial Network (GAN)-based image synthesis framework. We selectively extract the defect edges of the original image as well as the background texture information, and use them as network input through the spatially-adaptive (de)normalization (SPADE) module. This enriches the input information, thus significantly reducing the amount of training data for GAN network in image generation, and enhancing the background details as well as the defect boundaries in the generated images. Additionally, we introduce a novel generator loss term that balances the similarity and perceptual fidelity between synthetic and real images by constraining high-level features at different feature levels. This provides more valuable information for data augmentation in training object detection models using synthetic images. Our experimental results demonstrate the sophistication of the proposed image synthesis method and its effectiveness in data augmentation for steel strip surface defect detection tasks.