Robust Estimation for Non-parametric Families via Generative Adversarial Networks

Owing to their great success in unsupervised tasks, generative adversarial networks (GANs) are widely adopted for supervised (conditional) image-generation tasks such as in-painting. Unfortunately, designing an objective function using GANs is not trivial. For supervised tasks, the generator trained only with GAN loss does not yield the corresponding target for the given input because GAN is trained to match the data distribution, not to find the exact answer. Therefore, the loss function of a generator is often formulated by linearly combining supervised loss and GAN loss as similar to multi-task learning, expecting that each loss's weakness is complemented. Contrary to expectation, both losses cause a conflict in practice due to different optimum of each loss, yielding low objective and subjective image qualities. To address this problem, we empirically investigated the conflict caused by using a conventional GAN with pixel-wise losses; we then propose a novel (relativistic) accuracy-aware discriminator. Based on the proposed discriminator, we developed an accuracy-aware GAN (AAGAN) and proved its optimality under an ideal assumption. We then propose a relativistic accuracy-aware GAN (RAAGAN) by considering practical assumptions. Experimental results on supervised tasks demonstrated that the proposed schemes alleviated the competition between losses and outperformed conventional GANs in terms of both objective and subjective qualities.

Magnetic Resonance Imaging (MRI) is one of the most commonly used modalities in medical imaging, and one of the main limitations is its slow scanning and reconstruction speed. Recently, deep learning used for compressed sensing (CS) methods have been proposed to accelerate the acquisition by undersampling in the K-space and reconstruct images with neural networks. However, there are still some challenges remained: First, directly training networks based on L1/L2 distance to the target fully sampled images may lead to fuzzy reconstruction images because L1/L2 loss only enforces the overall image or patch similarity, but does not consider the local details such as anatomical sharpness. Second, Generative Adversarial Networks (GAN) can partially solve this problem. The undersampling image gets the latent space through the encoder, and the image is reconstructed by the decoder based on GAN loss, but it may generate unrealistic details by lacking of constraints in K-space domain. Third, most of the networks after training are fixed and have limited adaptation capability in the inference time, and the patient-specific information cannot be effectively used. To resolve these challenges, we proposed a new compressed sensing GAN reconstruction method, and there are two main contributions: (1) We proposed a encoder-decoder structure, which guided GAN optimization strategy data-consistency in latent space to improve the reconstruction quality such as preserving more local details and improving the anatomical sharpness while constraining GAN to follow the data distribution in the K-space to prevent the unrealistic details. (2) An online update strategy is used to find the best representation in the latent space for the underlying patient, and the reconstruct result can be further improved by incorporating the patient-specific information. Extensive experimental results show the effectiveness of our method. KeywordsMRI reconstructionData-consistencyLatent spaceOnline update

We explore why many recently proposed robust estimation problems are efficiently solvable, even though the underlying optimization problems are non-convex. We study the loss landscape of these robust estimation problems, and identify the existence of ’generalized quasi-gradients’. Whenever these quasi-gradients exist, a large family of no-regret algorithms are guaranteed to approximate the global minimum; this includes the commonly used filtering algorithm. For robust mean estimation of distributions under bounded covariance, we show that any first-order stationary point of the associated optimization problem is an approximate global minimum if and only if the corruption level $\epsilon < 1/3$. Consequently, any optimization algorithm that approaches a stationary point yields an efficient robust estimator with breakdown point $1/3$. With carefully designed initialization and step size, we improve this to $1/2$, which is optimal. For other tasks, including linear regression and joint mean and covariance estimation, the loss landscape is more rugged: there are stationary points arbitrarily far from the global minimum. Nevertheless, we show that generalized quasi-gradients exist and construct efficient algorithms. These algorithms are simpler than previous ones in the literature, and for linear regression we improve the estimation error from $O(\sqrt{\epsilon })$ to the optimal rate of $O(\epsilon )$ for small $\epsilon $ assuming certified hypercontractivity. For mean estimation with near-identity covariance, we show that a simple gradient descent algorithm achieves breakdown point $1/3$ and iteration complexity $\tilde{O}(d/\epsilon ^2)$.

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

<p>Appearance defects frequently occur during cigarette manufacturing due to production equipment or raw materials. Appearance defects significantly impact the quality of tobacco products. Since manual inspection cannot keep pace with the demands of high-speed production lines, rapid and accurate automated classification and detection are essential. Supervised learning is predominantly employed in research on automated classification of product quality appearance defects. However, supervised learning necessitates substantial labeled data for training, which is time-consuming to annotate and prone to errors. This paper proposes a self-supervised learning-based classification model for cigarette appearance defects. This is a generative adversarial network (GAN) model based on masked autoencoders (MAE), called MAE-GAN. First, this model combines MAE as a generator with a simple discriminator to form a generative adversarial network according to the principle of mask reconstruction in MAE. The generator reconstructs the images to learn their features. Second, the model also integrates MAE's loss function into the GAN's loss function. This lets the model focus on pixel-level losses during training. As a result, model performance is improved. Third, a Wasserstein GAN with gradient penalty (WGAN-GP) is added to stabilize the training process. In addition, this paper preprocesses cigarette images through segmentation and recombination. Neural networks typically accept images with the same width and height. Due to the narrow shape of cigarette images, if the image is directly transformed into a square and fed into a neural network, the details of the image will be severely lost. This paper segments the cigarette image into three parts and recombines them into images with similar length and width, greatly improving classification accuracy.</p>

Computed Tomography (CT) images play a crucial role in medical diagnosis and treatment planning. However, acquiring CT images can be difficult in certain scenarios, such as patients inability to undergo radiation exposure or unavailability of CT scanner. An alternative solution can be generating CT images from other imaging modalities. In this work, we propose a medical image translation pipeline for generating high-quality CT images from Positron Emission Tomography (PET) images using a Pix2Pix Generative Adversarial Network (GAN), which are effective in image translation tasks. However, traditional GAN loss functions often fail to capture the structural similarity between generated and target image. To alleviate this issue, we introduce a Multi-Scale Structural Similarity Index Measure (MS-SSIM) loss in addition to the GAN loss to ensure that the generated images preserve the anatomical structures and patterns present in the real CT images. Experiments on the ‘QIN-Breast’ dataset demonstrate that our proposed architecture achieves a Peak Signal-to-Noise Ratio (PSNR) of 17.70 dB and a Structural Similarity Index Measure (SSIM) of 42.51% in the region of interest.

Many applications require precise estimates of high-dimensional covariance matrices. The standard estimator is the sample covariance matrix, which is conceptually simple, fast to compute and has favorable properties in the limit of infinitely many observations. The picture changes when the dimensionality is of the same order as the number of observations. In such cases, the eigenvalues of the sample covariance matrix are highly biased, the condition number becomes large and the inversion of the matrix gets numerically unstable.
\nA number of alternative estimators are superior in the high-dimensional setting, which include as subcategories structured estimators, regularized estimators and spectrum correction methods. In this thesis I contribute to all three areas. 
\nIn the area of structured estimation, I focus on models with low intrinsic dimensionality. I analyze the bias in Factor Analysis, the state-of-the-art factor model and propose Directional Variance Adjustment (DVA) Factor Analysis, which reduces bias and yields improved estimates of the covariance matrix.
\nAnalytical shrinkage of Ledoit and Wolf (LW-Shrinkage) is the most popular regularized estimator. I contribute in three aspects: first, I provide a theoretical analysis of the behavior of LW-Shrinkage in the presence of pronounced eigendirections, a case of great practical relevance. I show that LW-Shrinkage does not perform well in this setting and propose aoc-Shrinkage which yields significant improvements. Second, I discuss the effect of autocorrelation on LW-Shrinkage and review the Sancetta-Estimator, an extension of LW-Shrinkage to autocorrelated data. I show that the Sancetta-Estimator is biased and propose a theoretically and empirically superior estimator with reduced bias. Third, I propose an extension of shrinkage to multiple shrinkage targets. Multi-Target Shrinkage is not restricted to covariance estimation and allows for many interesting applications which go beyond regularization, including transfer learning. I provide a detailed theoretical and empirical analysis.
\nSpectrum correction approaches the problem of covariance estimation by improving the estimates of the eigenvalues of the sample covariance matrix. I discuss the state-of-the-art approach, Nonlinear Shrinkage, and propose a cross-validation based covariance (CVC) estimator which yields competitive performance at increased numerical stability and greatly reduced complexity and computational cost. On all data sets considered, CVC is on par or superior in comparison to the regularized and structured estimators.
\nIn the last chapter, I conclude with a discussion of the advantages and disadvantages of all covariance estimators presented in this thesis and give situation-specific recommendations. In addition, the appendix contains a systematic analysis of Linear Discriminant Analysis as a model application, which sheds light on the interdependency between the generative model of the data and various covariance estimators.

Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains. They can also improve recognition despite the presence of domain shift or dataset bias: recent adversarial approaches to unsupervised domain adaptation reduce the difference between the training and test domain distributions and thus improve generalization performance. However, while generative adversarial networks (GANs) show compelling visualizations, they are not optimal on discriminative tasks and can be limited to smaller shifts. On the other hand, discriminative approaches can handle larger domain shifts, but impose tied weights on the model and do not exploit a GAN-based loss. In this work, we first outline a novel generalized framework for adversarial adaptation, which subsumes recent state-of-the-art approaches as special cases, and use this generalized view to better relate prior approaches. We then propose a previously unexplored instance of our general framework which combines discriminative modeling, untied weight sharing, and a GAN loss, which we call Adversarial Discriminative Domain Adaptation (ADDA). We show that ADDA is more effective yet considerably simpler than competing domain-adversarial methods, and demonstrate the promise of our approach by exceeding state-of-the-art unsupervised adaptation results on standard domain adaptation tasks as well as a difficult cross-modality object classification task.

Patch-Based DCNN Method for CBCT Image Enhancement

<p>High data collection costs and complicated data access regulations increase the demand for synthetic data. Generative Adversarial Networks (GANs) are a novel generative framework with great potential for high quality synthetic data generation. GANs formulate the true distribution of data implicitly, and the success of GANs are often measured based on the similarity of generated data to this true distribution. GANs were originally designed to work with continuous data. However, many important real-world datasets such as medical images involve discontinuous distributions. GAN training for discontinuous distributions is relatively more challenging, as the training procedure often suffers from instability and mode collapse issues. This dissertation focuses on designing novel GAN architectures to generate representative synthetic image data, and proposes new structures to alleviate GANs' mode collapse issue. As part of this thesis, novel applications of image data generation with GANs have been also investigated for important problems arising in the telecommunication industry and medical domain. Specifically, we first explore various GAN structures to generate engineered electromagnetic surfaces. We consider the continuous approximation of the data and explore the capabilities of feed-forward and convolutional GANs for synthetic data generation. Next, we introduce a novel GAN architecture to address the problem of mode collapse in GAN training. The proposed structure incorporates a third network that penalizes the generator for generating low diversity samples. Lastly, we study the challenging problem of object generation in 3D space using GANs, and we propose extensions to existing 3D GAN structures to generate connected 3D volumes. Additionally, we explore a more challenging version of this 3D volume generation problem by generating connected volumes packed with spheres. This research has applications in radiosurgery treatment planning, and the proposed 3D GAN structure can help generate rare, unseen 3D tumor volumes and information on how to treat these tumors. Accordingly, our analysis contributes to overcoming data scarcity issues (e.g., due to privacy considerations) for an important practical problem in the medical domain.</p>

Magnetic Particle Imaging (MPI), as an emerging non-invasive imaging technique, acquires high-resolution images by detecting the response of magnetic nanoparticles. Reconstruction based on System Matrix (SM) is an important component of MPI research. However, when scanning parameters, particle types, or environmental conditions change, SM requires repetitive and time-consuming calibration scans, significantly reducing its practicality. In this study, we propose a novel SM super-resolution recovery method, PGSM-net, based on Generative Adversarial Network (GAN) and Pyramid Attention Mechanism (PAM) to enhance SM calibration in MPI. The generator utilizes enhanced ResNet blocks and a PAM module for multi-scale feature extraction, attention weighting, and feature fusion to recover finer details from low-resolution inputs. The discriminator, an unpooled VGG network, assesses high-dimensional feature consistency to aid the generator in producing more realistic and higher-quality images. Our approach represents the first application of GAN and perceptual loss in MPI system matrix calibration. Experimental results show significant improvements in SM recovery quality and detail, highlighting its potential in MPI applications.

A well-known model, due to Fay and Herriot, for estimating small area (domain) means, μ i , is considered. Given μ i 's, it is assumed that the survey estimators, y i , are independent with means μ i and known variances Di, i = 1, …, t. Further, the μ i 's are assumed to be independent with means x′iβ and unknown variance A, where x i is a vector of benchmark variables related to μ i and β is a vector of regression parameters. An empirical best linear unbiased prediction (EBLUP) estimator or an empirical linear Bayes estimator, t i (Â, y), of μ i is obtained. It is shown that an estimator of mean squared error (MSE) of t t (Â, y), derived by Prasad and Rao under normality of μ i and y i given μ i , is robust with respect to nonnormality of the μ i 's. Specifically, it is shown that the Prasad–Rao estimator of MSE is correct to terms of order O(t −1) for large t, assuming only certain moment conditions on the μ i 's and normally distributed survey errors. Results of a simulation study on the accuracy of the estimator of MSE, under nonnormality of the μ i 's, are also presented.

Convolutional Neural Networks (CNNs) achieve excellent computer-assisted diagnosis with sufficient annotated training data. However, most medical imaging datasets are small and fragmented. In this context, Generative Adversarial Networks (GANs) can synthesize realistic/diverse additional training images to fill the data lack in the real image distribution; researchers have improved classification by augmenting data with noise-to-image (e.g., random noise samples to diverse pathological images) or image-to-image GANs (e.g., a benign image to a malignant one). Yet, no research has reported results combining noise-to-image and image-to-image GANs for further performance boost. Therefore, to maximize the DA effect with the GAN combinations, we propose a two-step GAN-based DA that generates and refines brain Magnetic Resonance (MR) images with/without tumors separately: (i) Progressive Growing of GANs (PGGANs), multi-stage noise-to-image GAN for high-resolution MR image generation, first generates realistic/diverse 256×256 images; (ii) Multimodal UNsupervised Image-to-image Translation (MUNIT) that combines GANs/Variational AutoEncoders or SimGAN that uses a DA-focused GAN loss, further refines the texture/shape of the PGGAN-generated images similarly to the real ones. We thoroughly investigate CNN-based tumor classification results, also considering the influence of pre-training on ImageNet and discarding weird-looking GAN-generated images. The results show that, when combined with classic DA, our two-step GAN-based DA can significantly outperform the classic DA alone, in tumor detection (i.e., boosting sensitivity 93.67% to 97.48%) and also in other medical imaging tasks.

Cross-modality image estimation can be performed using generative adversarial networks (GANs). To date, SPECT image estimation from another medical imaging modality using this technique has not been considered. We evaluate the estimation of SPECT from MRI and PET, and additionally assess the necessity for cross-modality image registration for GAN training. We estimated interictal SPECT from PET and MRI as a single-channel input, and as a multi-channel input to the GAN. We collected data from 48 individuals with epilepsy and converted them to 3D isotropic images for consistence across the modalities. Training and testing data were prepared in native and template spaces. The Pix2pix framework within the GAN network was adopted. We evaluated the addition of the structural similarity index metric to the loss function in the GAN implementation. Root-mean-square error, structural similarity index, and peak signal-to-noise ratio were used to assess how well SPECT images were able to be synthesised. High quality SPECT images could be synthesised in each case. On average, the use of native space images resulted in a 5.4% percentage improvement in SSIM than the use of images registered to template space. The addition of structural similarity index metric to the GAN loss function did not result in improved synthetic SPECT images. Using PET in either the single channel or dual channel implementation led to the best results, however MRI could produce SPECT images close in quality. Synthesis of SPECT from MRI or PET can potentially reduce the number of scans needed for epilepsy patient evaluation and reduce patient exposure to radiation.

The generative adversarial network (GAN) aims to approximate an unknown distribution via a parameterized neural network (NN). While GANs have been widely applied in reinforcement and semi-supervised learning as well as computer vision tasks, selecting their parameters often needs an exhaustive search and only a few selection methods can be proved to be theoretically optimal. One of the most promising GAN variants is the Wasserstein GAN (WGAN). Prior work on optimal parameters for WGAN is limited to the linear-quadratic-Gaussian (LQG) setting, where the NN is linear and the data is Gaussian. In this paper, we focus on the characterization of optimal WGAN parameters beyond the LQG setting. We derive closed-form optimal parameters for one-dimensional WGANs with non-linear sigmoid and ReLU activation functions. Extensions to high-dimensional WGANs are also discussed. Empirical studies show that our closed-form WGAN parameters have good convergence behavior with data under both Gaussian and Laplace distributions.