Sparse subspace clustering using square-root penalty

This paper investigates the application of unsupervised learning methods for computed tomography reconstruction. To motivate our work, we review several existing priors, namely the truncated Gaussian prior, the prior, the total variation prior, and the deep image prior (DIP). We find that DIP outperforms the other three priors in terms of representational capability and visual performance. However, the performance of DIP deteriorates when the number of iterations exceeds a certain threshold due to overfitting. To address this issue, we propose a novel method (MCDIP‐ADMM) based on multi‐code deep image prior (MCDIP) and plug‐and‐play alternative direction method of multipliers (ADMM). Specifically, MCDIP utilizes multiple latent codes to generate a series of feature maps at an intermediate layer within a generator model. These maps are then composed with trainable weights, representing the complete image prior. Experimental results demonstrate the superior performance of the proposed MCDIP‐ADMM compared to three existing competitors. In the case of parallel beam projection with Gaussian noise, MCDIP‐ADMM achieves an average improvement of 4.3 dB over DIP, 1.7 dB over ADMM DIP‐weighted total variation (WTV) and 1.2 dB over PnP‐DIP in terms of peak‐signal‐to‐noise ratio (PSNR). Similarly, for fan‐beam projection with Poisson noise, MCDIP‐ADMM achieves an average improvement of 3.09 dB over DIP, 1.86 dB over ADMM DIP‐WTV and 0.84 dB over PnP‐DIP in terms of PSNR.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

The Alternating Direction Method of Multipliers(ADMM) is an important method for machine learning. A large number of stochastic versions of ADMM continue to emerge. But almost all the algorithm focused on the Steepest Descent, which cause slow convergence rate. In our work, we propose a new stochastic alternative direction method of multipliers with conjugate gradient(SCG-ADMM), and prove its convergence rate in ergodic sense. Conjugate gradient descent has a better performance in traditional cases. Experiments on LASSO and Logistics Regression show that our proposed algorithm can achieve state-of-the-art performance in real applications. And we use the linear search method to replace the manual setting of the step length.

Recently, compressed sensing (CS) has been an effective data compression technique for telemonitoring of multichannel electroencephalogram (EEG) signals through wireless body-area networks. Most of the existing multichannel EEG CS methods ignore the noise or only consider the Gaussian noise. However, there are also some other types of noise, such as heavy-tailed impulsive noise. In this paper, to overcome the above mentioned problems, we propose a novel multichannel EEG CS method based on sparse and low rank representation in the presence of mixed noise (SLRMN), which can take both Gaussian noise and impulsive noise into consideration. Moreover, we develop the alternative direction method of multipliers (ADMM) to solve the proposed SLRMN. The experimental results demonstrate the advantage of the proposed SLRMN over state-of-the-art multichannel EEG CS methods in the presence of mixed noise.

In the process of signal and image acquisition and transmission, there is not only the additive white Gaussian noise interference, but also the non-Gaussian noise interference. The performance of a standard reconstruction algorithm, known as compressed sensing, can be significantly reduced if it is used to recover the signal affected by non-Gaussian noise. In this paper, LpLq-ADMM algorithm $(p \in(0,2), q \in(0,1))$ based on alternating direction method of multipliers is proposed to improve the reconstruction accuracy and the robustness of signal and image restoration in impulsive SαS (symmetric α-stable) noise. Lp-norm is used as the loss function term to enhance the data restoration ability in SαS noise and Lq-norm is used as the generalized non-convex penalty term to guarantee the sparsity of the signal. In order to efficiently solve the cost function model with minimum non-convex and nonsmooth, reduce the computational complexity of the objective function and improve the processing ability of high-dimensional data, in this paper, the alternating direction method of multipliers (ADMM) is used to solve the cost function model. The appropriate p and q values are selected via numerical simulation experiments to achieve the optimal reconstruction accuracy and robustness of sparse signals in SαS noise. The reconstruction probability of sparse signals and image recovery performance are analyzed under the optimal p and q values. The simulation results show that LpLq-ADMM algorithm has great noise suppression ability to SαS noise, and its Peak Signal to Noise Ratio (PSNR) and signal recovery probability is higher than other recovery algorithms.

Convex optimization is at the core of many of today's analysis tools for large datasets, and in particular machine learning methods. This thesis will develop approximate versions of the alternating directrion of multipliers (ADMM) for the general setting of minimizing the sum of two convex functions. The alternating direction method of multipliers is a form of augmented Lagrangian algorithm that has experienced a renaissance in recent years due to its applicability to optimization problems arising from ``big data'' and image processing applications, and the relative ease with which it may be implemented in parallel and distributed computational environments. There are two fundamental approaches for proving the convergence of the ADMM, each based on a different form of two-way emph{splitting}, that is, expressing a mapping as the sum of two simpler mappings. The first approach is based on Douglas-Rachford operator splitting theory, and yields considerable insight into the convergence of the ADMM. The second convergence proof approach is at its core based on the Lagrangian splitting analysis. We present three new approximate versions of ADMM based on both convergence analyses, all of which require only knowledge of subgradients of the subproblem objectives, rather bounds on the distance to the exact subproblem solution. One version, which applies only to certain common special cases, is based on combining the operator splitting analysis of the ADMM with a relative-error proximal point algorithm of Solodov and Svaiter. A byproduct of this analysis is a new, relative-error version of the Douglas-Rachford splitting algorithm for monotone operators. The other two approximate versions of the ADMM are more general and based on the Lagrangian splitting analysis of the ADMM: one uses a summable absolute error criterion, and the other uses a relative error criterion and an auxiliary iterate sequence. We experimentally compare our new algorithms to an essentially exact form of the ADMM and to an inexact form that can be easily derived from prior theory (but again applies only to certain common special cases). These experiments show that our methods can significantly reduce total computational effort when iterative methods are used to solve ADMM subproblems.

Reweighted sparse unmixing for hyperspectral images with noise level estimation

In this study, an effective robust PCA is developed for joint image alignment and recovery via L 2,1 norms and affine transformations. To alleviate the potential impacts of outliers, heavy sparse noises, occlusions, and illuminations, the L 2,1 norms along with affine transformations are taken into consideration. The determination of the parameters involved and the updating affine transformations is arranged in the form of a constrained convex optimization problem. To reduce the computation load, we also further decompose the error as sparse error and Gaussian noise; additionally, the alternating direction method of multipliers (ADMM) is considered to develop a new set of recursive equations to update the optimization parameters and the affine transformations iterative. The convergence of the derived updating equation is explained as well. Conducted simulations illustrate that the new method is superior to the baseline works in terms of precision on some public databases.

Infrared (IR) images are essential to improve the visibility of dark or camouflaged objects. Object recognition and segmentation based on a neural network using IR images provide more accuracy and insight than color visible images. But the bottleneck is the amount of relevant IR images for training. It is difficult to collect real-world IR images for special purposes, including space exploration, military and fire-fighting applications. To solve this problem, we created color visible and IR images using a Unity-based 3D game editor. These synthetically generated color visible and IR images were used to train cycle consistent adversarial networks (CycleGAN) to convert visible images to IR images. CycleGAN has the advantage that it does not require precisely matching visible and IR pairs for transformation training. In this study, we discovered that additional synthetic data can help improve CycleGAN performance. Neural network training using real data (N = 20) performed more accurate transformations than training using real (N = 10) and synthetic (N = 10) data combinations. The result indicates that the synthetic data cannot exceed the quality of the real data. Neural network training using real (N = 10) and synthetic (N = 100) data combinations showed almost the same performance as training using real data (N = 20). At least 10 times more synthetic data than real data is required to achieve the same performance. In summary, CycleGAN is used with synthetic data to improve the IR image conversion performance of visible images.

This paper studies the hyperspectral image (HSI) denoising problem under the assumption that the signal is low in rank. In this paper, a mixture of Gaussian noise and sparse noise is considered. The sparse noise includes stripes, impulse noise, and dead pixels. The denoising task is formulated as a low-rank tensor recovery (LRTR) problem from Gaussian noise and sparse noise. Traditional low-rank tensor decomposition methods are generally NP-hard to compute. Besides, these tensor decomposition based methods are sensitive to sparse noise. In contrast, the proposed LRTR method can preserve the global structure of HSIs and simultaneously remove Gaussian noise and sparse noise.The proposed method is based on a new tensor singular value decomposition and tensor nuclear norm. The NP-hard tensor recovery task is well accomplished by polynomial time algorithms. The convergence of the algorithm and the parameter settings are also described in detail. Preliminary numerical experiments have demonstrated that the proposed method is effective for low-rank tensor recovery from Gaussian noise and sparse noise. Experimental results also show that the proposed LRTR method outperforms other denoising algorithms on real corrupted hyperspectral data.

This paper proposes a Variational Mode Decomposition (VMD) based approach for enhancement of speech signals distorted by white Gaussian noise. VMD is a data adaptive method which decomposes the signal into intrinsic mode functions (IMFs) by using the Alternating Direction Method of Multipliers (ADMM). Each IMF or mode will contain a center frequency and its harmonics. This paper tries to explore VMD as a Speech enhancement technique. In the proposed method, the noisy speech signal is decomposed into IMFs using VMD. The noisy IMFs are enhanced using two methods; VMD based wavelet shrinkage (VMD-WS) and VMD based MMSE log STSA (VMD-MMSE). The speech signal distorted with different noise levels are enhanced using the VMD based methods. The level of noise reduction and speech signal quality are measured using the objective quality measures.

Distributed optimization has gained popularity in many areas because it can to cope with the increasing volume of data, and the corresponding demand for computational resources. A popular distributed solver is the alternating direction method of multipliers (ADMM). Fully decentralized (D) ADMM arises when node-to-node communications have to abide by the underlying network connectivity. DADMM’s convergence however, slows down as the network diameter grows large. To deal with this challenge, the recently proposed hybrid (H) ADMM provides considerable performance boost over DADMM by exploiting local topology information. But HADMM only applies to unweighted graphs. The present contribution develops a weighted hybrid (WH) consensus-based ADMM approach that can deal with weighted graphs. The resultant scheme further improves the performance of HADMM through graph-aware weight tuning. Theoretical analysis offers convergence guarantees and establishes linear convergence rate, while numerical tests on various graphs demonstrate the WHADMM merits.

Parameter estimation of damped exponential signals has wide applications including fault detection and system parameter identification, etc. However, existing methods for estimating parameters of damped exponentials are either sensitive to noise or restricted to dealing with a certain type of noise such as Gaussian noise. In this paper we aim to estimate parameters of damped exponentials from contaminated signal, i.e., a mixture of damped exponentials, random Gaussian noise, and spike interference. We propose two robust approaches, a convex one solved by the alternating direction method of multipliers (ADMM) and a non-convex one solved by coordinate descent, to recovering a low-rank Hankel matrix of damped exponentials from noisy measurements for further parameter estimation using the matrix pencil technique. Numerical experiments show that our proposed methods outperform classical ones in detecting small damped fault signatures from noisy measurements. While the convex approach is amenable to theoretical analysis and global convergence guarantees, the non-convex one exhibits more robustness and computational efficiency.

Assessing the identifiability of model selection frameworks for the prediction of patient outcomes in the clinical breast cancer setting

Poisson noise is an important type of electronic noise that is present in a variety of photon-limited imaging systems. Different from the Gaussian noise, Poisson noise depends on the image intensity, which makes image restoration very challenging. Moreover, complex geometry of images desires a regularization that is capable of preserving piecewise smoothness. In this paper, we propose a Poisson denoising model based on the fractional-order total variation (FOTV). The existence and uniqueness of a solution to the model are established. To solve the problem efficiently, we propose three numerical algorithms based on the Chambolle-Pock primal-dual method, a forward-backward splitting scheme, and the alternating direction method of multipliers (ADMM), each with guaranteed convergence. Various experimental results are provided to demonstrate the effectiveness and efficiency of our proposed methods over the state-of-the-art in Poisson denoising.