ContourGAN: Image contour detection with generative adversarial network

The generative model is a very important type of model in the field of artificial intelligence in recent years. Such models can comprehend the data through the neural network, and then create data according to the probability distribution of the input data, predict the results according to the data characteristics. The whole processing process is intelligent. At present, two typical applications of generative model are Generative Adversarial Networks (GAN) model and Encoder-Decoder model, which have strong ability to generate image data. In the model of GAN, the generator simulates real data, and the discriminator judges the authenticity of the samples. Its goal is to train a generator to fit the real data perfectly, so that the discriminator cannot distinguish. In the Encoding-Decoding model, it can be understood as a process of Encoding→intermediate vector→Decoding, It is suitable for processing one kind of data to generate another kind of data with the same probability distribution as the original data. Since most of the data features are intermingled, they are encoded in a complex and disorderly way. But if these features are extractable, it shows that these features are interpretable, and it will be easier to use these features for coding. Based on this situation, some research results have been produced by incorporating the Encoder- Decoder into GAN. This paper systematically analyzes and summarizes the basic concepts and theories of GAN and Encoder-Decoder, as well as their respective advantages and disadvantages. On this basis, by combing the related work of the two types of models, the main technical routes of the three types of GAN based on the Encoder Decoder are summarized, the techniques and theories including variational inference, energy function and correlation transformation of different distribution data are summarized. Finally, the GAN based on the Encoder-Decoder is summarized.

Segmentation of breast cancer tumor plays an important role in identifying the location of the tumor, to know the shape of tumor and hence the stage of breast cancer. This paper deals with the segmentation of tumor from whole mammographic mass images using Generative Adversarial Network (GAN). A mini dataset was considered with mammograms and their corresponding ground truth images. Pre-processing like image format conversion, enhancement, pectoral muscle removal and resizing was performed on raw mammogram images. GANs have two neural nets called generative and discriminative networks that compete against each other to obtain the segmentation output. PIX2PIX is a conditional GAN variant which has U-Net as the Generator network and a simple deep neural net as the discriminator. The input to the network was pair of pre-processed mass image and the associated ground truth. A binary image with highlighted tumor was obtained as output. The performance of GAN was evaluated by plotting Generator and discriminator loss. The segmented output was compared with corresponding ground truth. Metrics like Jaccard index, Jaccard distance and Dice-coefficient were calculated. A Dice-coefficient and Jaccard index of 90% and 88.38% was achieved. In future, higher accuracy could be achieved by involving larger dataset to make the system robust.

Generative Adversarial Network is the topic of interest in today’s research in the field of image processing and computer vision. A basic GAN model was introduced by Ian Goodfellow et al. in 2014. After that advancement in the field of research in GAN models has been application specific. In computer vision and image to image translation GANs are playing very effective role either in the case of face detection and recognition or in image resolution enhancement and image augmentation. This paper represents a concise overview of various GAN models along with their features and applications. Pix2Pix and conditional GAN models work upon paired datasets while other models like cycle GAN, discover GAN, dual GAN, info GAN, deep convolutional GAN etc. work upon unpaired datasets. Various image datasets which are commonly used for training of generator and discriminator networks are also discussed in this paper. Since partial mode collapse is a common problem to occur during training process for all models, therefore various normalization techniques are also preferred during the training of generator and discriminator networks. As the advancements in GAN models are increasing at a very fast rate, soon these models will also be preferred in commercial applications.

Purpose:Image guided radiotherapy provides images not only for patientpositioning but also for online adaptive radiotherapy. Accurate delineationof organs-at-risk (OARs) on Head and Neck (H&N) CT and MR images isvaluable to both initial treatment planning and adaptive planning, butmanual contouring is laborious and inconsistent. A novel method based on thegenerative adversarial network (GAN) with shapeconstraint (SC-GAN) is developed for fullyautomated H&N OARs segmentation on CT and low field MRI.Methods and material:A deep supervised fully convolutional DenseNet is employed as thesegmentation network for voxel-wise prediction. A CNN based discriminatornetwork is then utilized to correct predicted errors and image-levelinconsistency between the prediction and ground truth. An additional shaperepresentation loss between the prediction and ground truth in the latentshape space is integrated into the segmentation and adversarial lossfunctions to reduce false positivity and constrain the predicted shapes. Theproposed segmentation method was first benchmarked on a public H&N CTdatabase including 32 patients, and then on 25 0.35T MR images obtained froman MR guided radiotherapy system. The OARs include brainstem, opticalchiasm, larynx (MR only), mandible, pharynx (MR only), parotid glands (bothleft and right), optical nerves (both left and right), and submandibularglands (both left and right, CT only). The performance of the proposedSC-GAN was compared with GAN alone and GAN with the shape constraint (SC)but without the Densenet (SC-GAN-ResNet) to quantify the contributions ofshape constraint and DenseNet in the deep neural network segmentation.Results:The proposed SC-GAN slightly but consistently improve thesegmentation accuracy on the benchmark H&N CT images compared with ourprevious deep segmentation network, which outperformed other publishedmethods on the same or similar CT H&N dataset. On the low field MRdataset, the following average Dice’s indices were obtained usingimproved SC-GAN: 0.916 (Brainstem), 0.589 (Optical chiasm), 0.816(Mandible), 0.703 (Optical nerves), 0.799 (Larynx), 0.706 (Pharynx), and0.845 (Parotid glands). The average surface distances ranged from 0.68mm(Brainstem) to 1.70mm (Larynx). The 95% surface distance ranged from 1.48mm(Left optical nerve) to 3.92mm (Larynx). Compared with CT, using 95% surfacedistance evaluation, the automated segmentation accuracy is higher on MR forthe brainstem, optical chiasm, optical nerves and parotids, and lower forthe mandible. The SC-GAN performance is superior to SC-GAN-ResNet, which ismore accurate than GAN alone on both the CT and MR datasets. Thesegmentation time for one patient is 14 seconds using a single GPU.Conclusion:The performance of our previous shape constrained fully convolutionalneural networks for H&N segmentation is further improved byincorporating GAN and DenseNet. With the novel segmentation method, weshowed that the low field MR images acquired on a MR guided radiationradiotherapy system can support accurate and fully automated segmentation ofboth bony and soft tissue OARs for adaptive radiotherapy.

As a promising generative modeling method, Generative Adversarial Networks are a deep-learning-based generative model, in which two networks, namely the generative network and the discriminant network, contest with each other in a game during which the generator “learns” and get trained to be able to fool the discriminator of believing a specific image is (superficially) authentic. It is widely used to generate pictures based on existing data. Based on the MNIST data set, this paper studies and analyzes the application of Generative Adversarial Networks in the generation of handwritten digital images. In addition, by studying the Deep Convolutional Generative Adversarial Network and the Conditional Generation Adversarial Network model, we compared the connections and differences between these three models in training the MNIST dataset to generate handwritten digits algorithms, and gave an optimization method for generating handwritten digits.

A survey on handwritten mathematical expression recognition: The rise of encoder-decoder and GNN models

The science of deep learning has undergone a revolution thanks to the invention of generative adversarial networks (GANs), a kind of deep learning design that can produce synthetic data that is equivalent to real-world data. GANs have recently been employed for a number of images processing operations, including photo colorization. The storing of grayscale photos, which require less storage space and can then be colored, and the restoration of old photographs are two of the key uses for image colorization. Although there are numerous color choices that can be employed with a specific grayscale image, this problem is challenging. Deep learning has been used in recent advances to make an effort to overcome this issue. In addition to the generator network, the Pix2Pix architecture also has a discriminator network that tries to tell bogus from real color images. Many datasets, including the CIFAR-10 and COCO datasets, have shown that the suggested method produces high-quality colorized images that closely resemble the ground truth images. Overall, this study shows how well GANs–and specifically the Pix2Pix architecture–can be used to colorize images. According to the research, this technology might be applied to a number of activities, including the processing of images and videos, digital restoration, and aesthetic representation.

In recent years, many crimes use technology to generate someone's face which has a bad effect on that person. Generative adversarial network is a method to generate fake images using discriminators and generators. Conventional GAN involved binary cross entropy loss for discriminator training to classify original image from dataset and fake image that generated from generator. However, use of binary cross entropy loss cannot provided gradient information to generator in creating a good fake image. When generator creates a fake image, discriminator only gives a little feedback (gradient information) to generator update its model. It causes generator take a long time to update the model. To solve this problem, there is an LSGAN that used a loss function (least squared loss). Discriminator can provide astrong gradient signal to generator update the model even though image was far from decision boundary. In making fake images, researchers used Least Squares GAN (LSGAN) with discriminator-1 loss value is 0.0061, discriminator-2 loss value is 0.0036, and generator loss value is 0.575. With the small loss value of the three important components, discriminator accuracy value in terms of classification reaches 95% for original image and 99% for fake image. In classified original image and fake image in this studyusing a supervised contrastive loss classification model with an accuracy value of 99.93%.

Many machine learning models require huge datasets to get trained, to make correct predictions, or to increase their accuracy, that's where generative adversarial network (GAN) comes into the picture. GAN is a deep learning model, which can be used to create artificial data. It consists of two neural networks that cooperate in a way that resembles a game: a generator network and a discriminator network. Within the suggested model in this work, a dense motion network-based first-order motion model for image animation is developed. Here a trained GAN extracts the face landmarks from the driving video and develops the embedding model to create the synthesis video using the dedicated module to prepare the Deepfakes, employing key point detectors as a baseline. Lastly, an approach that makes use of dense motion networks to increase the effectiveness of a collection of GAN generators is provided. With the help of the sequel driving combination of driving video with the source image, the given results produce the augmented animation video. Hence, it is tried to implement a lighter model which consists of modules, with the approximate same accuracy when compared with other implementations of GAN. This work has a wide range of applications, including doubling dataset counts with a small number of sources, creating real-time backgrounds and characters for the gaming and animation industries using CG platforms, translating clothes, predicting videos, creating 3D objects, etc.

Chapter 14 - Generative adversarial network for video analytics

- Deepfakes are artificially generated multimedia content that can convincingly mimic real human faces and voices using advanced AI techniques such as Generative Adversarial Networks (GANs). This poses serious ethical, social, and security challenges in digital communication. To address this issue, the proposed project presents a Multimodal Deepfake Detection System that integrates image, video, and audio analysis pipelines within a unified framework. The system employs Efficient Net-based CNN for image forgery detection, CNN combined with Bi-LSTM for temporal video analysis, and a 1D-CNN with LSTM for detecting manipulated or cloned audio. The predictions from these modalities are combined using a Fuzzy Fusion Engine, which intelligently weights each confidence score to produce a final verdict with high accuracy and interpretability. The model is trained using public Deepfake datasets such as Face Forensics++, Celeb-DF, and DFDC, with binary cross-entropy loss, data augmentation, and early stopping to ensure stable convergence and better generalization. The trained models are deployed on Hugging Face Spaces, while the web interface is hosted on Vercel, enabling real-time Deepfake detection for users through a browser interface. This approach enhances detection accuracy by leveraging multimodal evidence (visual, temporal, and auditory), improves generalization across datasets, and provides an explainable and efficient solution to combat the growing threat of Deepfakes. Key Words: Deepfake Detection, Multimodal Learning, Bi-LSTM, Fuzzy Fusion, Convolutional Neural Network (CNN), Face Forensics++, Celeb DF, DFDC, Machine Learning, Real-Time Detection, Hugging Face.

The standard evaluation metric of automatic speech recognition (ASR) is the word error rate (WER), which measures the dissimilarity between recognized word sequences and their ground truth. Many training algorithms designed to reduce sequence-level errors such as WER have been proposed for hidden Markov model (HMM)-based ASR, e.g., state-level minimum Bayes risk (sMBR). However, these approaches cannot be used directly for encoder-decoder model based end-to-end ASR, because the encoder-decoder model employs very different mechanisms from HMM-based approaches. In this paper, we propose a new method for optimizing the encoder-decoder model based on a sequence-level evaluation metric. Since the WER is not directly differentiable, we adopt a policy gradient objective function to train the encoder-decoder model, which enables us to minimize the expected WER of the model predictions. This training method employs the scoring of multiple hypotheses as in the decoding stage while usual cross entropy training uses only the ground truth. Therefore, we can expect it to improve the decoding results of the encoder-decoder model. We perform experiments using the Tedlium corpus to demonstrate the potential of our proposed method for improving the recognition performance of the encoder-decoder model.

Generative adversarial networks (GANs) create images by pitting a generator (G) against a discriminator (D) network, aiming to find a balance between the networks. However, achieving this balance is difficult because G is trained based on just one value representing D’s prediction, and only D can access image features. We introduce a novel approach for training GANs using explainable artificial intelligence (XAI) to enhance the quality and diversity of generated images in histopathological datasets. We leverage XAI to extract feature information from D and incorporate it into G via the loss function, a unique strategy not previously explored in this context. We demonstrate that this approach enriches the training with relevant information and promotes improved quality and more variability in the artificial images, decreasing the FID by up to 32.7% compared to traditional methods. In the data augmentation task, these images improve the classification accuracy of Transformer models by up to 3.81% compared to models without data augmentation and up to 3.01% compared to traditional GAN data augmentation. The Saliency method provides G with the most informative feature information. Overall, our work highlights the potential of XAI for enhancing GAN training and suggests avenues for further exploration in this field.

Generative adversarial network (GAN) has been widely applied to produce super-resolution (SR) image with real perception and texture details. In most existing SR approaches, the training objective typically measures a pixel-wise average distance between the SR and high-resolution (HR) images. However, as the degradation function of different images from HR to low resolution (LR) is generally different, optimizing such metrics often leads to certain unpleasant artificial traces. Unlike the prevalent GAN inversion methods that require expensive image-specific optimization at runtime, we present an alternative formulation by directly leveraging latent representation produced by a pretrained AutoEncoder. We call this improved method reduce dimension super-resolved GAN (RD-SRGAN). RD-SRGAN first obtains the latent feature representation of LR image by a pretrained AutoEncoder as input to the generator network. This process not only reduces noise effects but also decreases the overall computational complexity. On the other hand, the residual between the ground truth and the produced images replaces the produced images as input to the discriminator network, and a 2D zero mean Gaussian noise with controllable low variance replaces the real images as another input to the discriminator network. By leveraging the feature representation and properties of the 2D zero mean Gaussian noise, we restrict the optimization space to produce an SR image. Therefore, the residual of the generated SR images tend to approximates to a Gaussian noise, which introduces useless deviation information as little as possible. Experimental results show that RD-SRGAN can benefit from these strategies and achieve improved fidelity and naturalness comparison to existing methods. Switching the pretrained AutoEncoder allows the method to deal with images from diverse categories, e.g., remote sensing satellite imaging, medical imaging, and astronomy.

Extraction of the luminal contours from the intravascular ultrasound (IVUS) images is very important to analysis and diagnosis of coronary heart disease. Manual processing of large IVUS data is quite tedious and time consuming. This paper presented an algorithm for automatic detection of the luminal contours in intravascular ultrasound images, based on fuzzy clustering and snakes. To solve the difficulty of automatic contour initialization, this paper used fuzzy clustering and spline interpolation to obtain the initial contour. First, fuzzy clustering was used to detect the luminal contours on the multiple longitudinal images. Then, luminal contour points were transformed into the individual transversal images. Those luminal contour points were spline-interpolated on these transversal images. The spline-interpolated contour was used as the initial contour of snakes. We evaluated automatically detection method based on the average contours obtained from expert manual segmentation as the ground truth, and the results had demonstrated that our method was accurate and efficient.