Adaptive multi-text union for stable text-to-image synthesis learning

Synthesizing high-resolution realistic images from text description using one iteration Generative Adversarial Network (GAN) is difficult without using any additional techniques because mostly the blurry artifacts and mode collapse problems are occurring. To reduce these problems, this paper proposes an Iterative Generative Adversarial Network (iGAN) which takes three iterations to synthesize high-resolution realistic image from their text description. In the \(1^{st}\) iteration, GAN synthesizes a low-resolution \(64 \times 64\) pixels basic shape and basic color image from the text description with less mode collapse and blurry artifacts problems. In the \(2^{nd}\) iteration, GAN takes the result of the \(1^{st}\) iteration and text description again and synthesizes a better resolution \(128 \times 128\) pixels better shape and well color image with very less mode collapse and blurry artifacts problems. In the last iteration, GAN takes the result of the \(2^{nd}\) iteration and text description as well and synthesizes a high-resolution \(256 \times 256\) well shape and clear image with almost no mode collapse and blurry artifacts problems. Our proposed iGAN shows a significant performance on CUB birds and Oxford-102 flowers datasets. Moreover, iGAN improves the inception score and human rank as compare to the other state-of-the-art methods.

AraBERT and DF-GAN fusion for Arabic text-to-image generation

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.

Advances in technology have led to the development of methods that can create desired visual multimedia. In particular, image generation using deep learning has been extensively studied across diverse fields. In comparison, video generation, especially on conditional inputs, remains a challenging and less explored area. To narrow this gap, we aim to train our model to produce a video corresponding to a given text description. We propose a novel training framework, Text-to-Image-to-Video Generative Adversarial Network (TiVGAN), which evolves frame-by-frame and finally produces a full-length video. In the first phase, we focus on creating a high-quality single video frame while learning the relationship between the text and an image. As the steps proceed, our model is trained gradually on more number of consecutive frames. This step-by-step learning process helps stabilize the training and enables the creation of high-resolution video based on conditional text descriptions. Qualitative and quantitative experimental results on various datasets demonstrate the effectiveness of the proposed method.

Text to Image Generation using GAN

Segmentation of breast cancer tumor plays an important role in identifying the location of the tumor, to know the shape of tumor and hence the stage of breast cancer. This paper deals with the segmentation of tumor from whole mammographic mass images using Generative Adversarial Network (GAN). A mini dataset was considered with mammograms and their corresponding ground truth images. Pre-processing like image format conversion, enhancement, pectoral muscle removal and resizing was performed on raw mammogram images. GANs have two neural nets called generative and discriminative networks that compete against each other to obtain the segmentation output. PIX2PIX is a conditional GAN variant which has U-Net as the Generator network and a simple deep neural net as the discriminator. The input to the network was pair of pre-processed mass image and the associated ground truth. A binary image with highlighted tumor was obtained as output. The performance of GAN was evaluated by plotting Generator and discriminator loss. The segmented output was compared with corresponding ground truth. Metrics like Jaccard index, Jaccard distance and Dice-coefficient were calculated. A Dice-coefficient and Jaccard index of 90% and 88.38% was achieved. In future, higher accuracy could be achieved by involving larger dataset to make the system robust.

The effect of loss function on conditional generative adversarial networks

Distributed optical fiber acoustic sensing (DAS) is a new and rapid-developing detection technology in seismic exploration. Unfortunately, due to the weak energy of scattered optical signals and the inferior coupling between DAS cable and receiving interface, the seismic data received by DAS are often characterized by low signal-to-noise ratio (SNR); this low SNR is likely to affect some subsequent analysis, such as inversion, imaging, and interpretation. In addition, the noise caused by the inferior coupling is a new kind of noise not presented on conventional seismic data. To enhance the SNR of DAS seismic data and suppress the DAS noise effectively, we propose a convolutional adversarial denoising network (CADN) based on the basic strategy of generative adversarial network (GAN) and the usage of a denoiser to replace the original generator in GAN. In CADN, the performance of denoiser is significantly strengthened via its own mean square error (MSE) loss and the adversarial loss between it and the discriminator. To balance the two losses and thus ensure the optimization of denoiser, we construct a novel loss function, where the optimal ratio of MSE and adversarial losses is determined by quantifying the denoising performance. Both real and synthetic examples are included to testify the denoising performance of CADN. Experimental results have demonstrated that CADN can suppress most of the DAS noise and enhance the SNR of DAS seismic data; also, it can recover the effective signals completely, even the extremely weak effective signals reflected by deep layers.

Generative adversarial networks (GANs) can generate sentences that imitate real language features from discrete spaces, and the implicit expression space learned by training can continuously generate credible sentences. We solve the problem of face image restoration of different genders, different skin colors, and different hair colors through a method of generating face images, in which the face image generated by text description has a better generation effect. We propose a face image generation method (T-GAN) based on generating confrontational text. Due to the particulars of a face, a Word-long short-term memory model is used to extract the text features corresponding to the face. Then combined with the idea of generating a game against “game,” a counter-text network is created, next a third type of input is added to the discriminator, and the real image with unmatched text is composed to strengthen the discriminator’s training effect and force discrimination. The device determines whether the generated image conforms to the text description, so the discriminator can better learn the relationship between the text description and the image content. The experimental results show that, compared with other image generation methods, the face image generated by the proposed method can give high quality generated images with a better effect and less distortion, which can better maintain the original attributes of the image, has the use value, and increases the reduction degree by 3.84%.

In practical application, the research of synthetic aperture radar (SAR) target recognition has fallen into a bottle-neck due to the lack of samples. Data argumentation methods based on generative adversarial networks (GAN) have received widespread attention in solving this type of few-shot sample problem. However, the generated images suffer from various shortcomings, such as lack of diversity, low signal-to-noise ratio, blur, etc. In this article, the VAE-WGANGP is proposed, which combines GAN and variational autoencoder (VAE) to alleviate these shortcomings. The innovations of this paper are as follows: firstly, the generator of GAN is replaced with VAE, which constructs an asymmetric network ensuring the stability of GAN training; secondly, the asymmetric loss function is composed of four parts, including reconstruction loss, divergence loss, adversarial loss, and gradient penalty. In this way, the problem of gradient explosion or gradient disappearance is alleviated. The experimental results with the MSTAR dataset show that the images generated by the proposed model outperform the advanced technology with many similar deep features and achieve significant improvement in the target recognition accuracy rate.

Text-to-Face synthesis has considerable challenges and potentials in the field of public safety. Compared with the Text-to-Image synthesis models, the text descriptions of facial features are more complex and diverse. For the text embedding, most of the previous Text-to-Face synthesis models only deal with a single sentence containing several features of face images, and the generated images are vague and lack of details. In this paper, a novel Progressive Text-to-Face synthesis with Generative Adversarial Network (PFGAN) is proposed to generate natural face images from text descriptions. Firstly, a new text encoding method Convolution-Deconvolution Word Embedding LSTM (CDWE-BLSTM) is leveraged as the text encoder, which tackles more complex sentences and improves the accuracy of text encoding. Secondly, the PFGAN is composed of multiple generators and discriminators arranged in a tree-like structure. Furthermore, face images at multiple scales are progressively generated from different branches of the tree, corresponding to the same descriptions. images at multiple scales corresponding to the same scene are generated from different branches of the tree. By comparing with three existing Text-to-Face synthesis methods, extensive experiments demonstrate that the proposed PFGAN is very competitive in the IS (Inception Scores), FID (Frechet Inception Distance) and resolution of the generated face images.

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

Network architecture search achieves state-of-the-art results in various tasks such as classification and semantic segmentation. Recently, a reinforcement learning-based approach has been proposed for generative adversarial networks (GANs) search. In this work, we propose an alternative strategy for GAN search by using a proxy task instead of common GAN training. Our method is called differentiable efficient generator search, which focuses on efficiently finding the generator in the GAN. Our search algorithm is inspired by the differential architecture search strategy and the global latent optimization procedure. This leads to both an efficient and stable GAN search. After the generator architecture is found, it can be plugged into any existing framework for GAN training. For consistency-term GAN, which we use in this work, the new model outperforms the original inception score results by 0.25 for CIFAR-10.

Synthesis of a realistic image from matching visual descriptions provided in the textual format is a challenge that has attracted attention in the recent research community in the field of artificial intelligence. Generation of the image from given text input is a problem, where given a text input, an image which matches text description must be generated. However, a relatively new class of convolutional neural networks referred to as generative adversarial networks (GANs) has provided compelling results in understanding textual features and generating high-resolution images. In this work, the main aim is to generate an automobile image from the given text input using generative adversarial networks and manipulate automobile colour using text-adaptive discriminator. This work involves creating a detailed text description of each image of a car to train the GAN model to produce images. KeywordsGenerative adversarial networksGeneratorDiscriminatorGANSText to image synthesis

Automatic image annotation (AIA) has been adopted in different applications such as image retrieval and classification. Deep Learning is used in AIA to extract image features and then convert these features into text descriptions and labels. However, conventional AIA models that employ deep learning methods suffer from various shortcomings, such as poor annotation performance. This work proposes an AIA model based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transfer learning. GANs have attracted a lot of interest because of its ability to generate data without explicitly using probability density. Thus, it has proven its usefulness in image annotation and image augmentation. In this work, an Auxiliary classifier-GAN (ACGAN) has been used, where the discriminator predicts the class of an image rather than taking it as a given input; therefore, the stabilization of the training stage is ensured, and the generation of high-quality images is provided. Transfer learning is also used to enhance the performance of the classification. The proposed model outperforms the best state-of-the-art models in terms of MiAP, F-measure and error rate using ImageClef, ESPGame and IAPR-TC12 datasets.