Multi-view Representative and Informative Induced Active Learning

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

Transient stability assessment (TSA) based on deep learning has recently attracted broad attention. The offline training of deep learning model typically requires a large number of labeled samples, and the training process is very time-consuming. Moreover, it is difficult to ensure that the trained model is suitable for all the operation conditions of power systems. In order to dramatically reduce the computation cost, and further enhance the adaptability of the model, an innovative method is proposed based on the active transfer learning (TL). First, the deep belief network (DBN) is designed as the base model and trained with a large number of labeled samples to obtain better evaluation performance for TSA. Second, when the topologies or operation conditions change substantially, active learning with information entropy is exploited to select the minimum number of the most informative samples, which effectively reduces the time of samples generation. Third, Maximum Mean Discrepancy (MMD) is calculated to select different transfer schemes. Only the last BP layer needs to be fine-tuned for the scenarios with a low MMD, but for those with high MMD, the entire network needs to be fine-tuned. The transfer time is greatly shortened on the premise of ensuring the transfer effect. The comprehensive experimental results of three power systems illustrate the effectiveness of the proposed method. Note to Practitioners—This research is motivated by automatic adaptive assessment of power system transient stability. The main objective is to realize online fast and continuous prediction with limited samples. This kind of demand is very normal in the industrial systems where it is expensive, time-consuming and difficult to obtain labeled samples. This paper proposes an active transfer learning method to select critical samples and update the deep learning model, which greatly reduces the time of samples generation and improves the updating efficiency of model. The proposed method is also applicable to make classification prediction or to address regression problems of industrial scenarios based on deep learning models.

The alternating direction method of multipliers (ADMM) with over-relaxation and the accelerated ADMM are two effective methods to improve the convergence rate of ADMM. In this paper, an efficient linear programming (LP) decoder based on accelerated ADMM for low-density parity-check (LDPC) codes is proposed. Then it is shown that the ADMM update steps for decoding LDPC codes using the accelerated ADMM and the over-relaxed ADMM are quite similar. Therefore, it is impossible to obtain a more efficient ADMM decoder by combining the two improved ADMM methods. Simulation results also validate that the average number of iterations for LDPC decoding using the accelerated ADMM and the over-relaxed ADMM are close to each other.

This paper discusses about a sparsity based algorithm used for Hyperspectral Image (HSI) classification where the test pixel vectors are sparsely represented as the linear combination of a few number of training samples from a well-organised dictionary matrix. The sparse vector is obtained using Basis Pursuit (BP) which is a constrained l4 minimization problem. This problem is solved by using a simple and powerful iterative algorithm known as Alternating Direction Method of Multipliers (ADMM) which significantly reduces the computational complexity of the problem and thereby speeds up the convergence. The classification accuracy is considerably improved by including efficient preprocessing techniques to remove the unwanted information (noise) present in Hyperspectral images. This paper uses a fast and reliable denoising technique based on Legendre Fenchel Transformation (LFT) to effectively denoise each band of HSI prior to ADMM based classification (proposed method). A comparison of proposed technique with one of the convex optimization tools namely, CVX is given to exhibit the fast convergence of the former method. The experiment is performed on standard Indian Pines dataset captured using AVIRIS sensor. The potential of the proposed method is illustrated by analyzing the classification indices obtained with and without applying any preprocessing methods. With only 10% training set, an overall accuracy of 96.76% is obtained for the proposed method at a much faster rate compared to computation time taken by CVX solver.

Convex optimization is at the core of many of today's analysis tools for large datasets, and in particular machine learning methods. This thesis will develop approximate versions of the alternating directrion of multipliers (ADMM) for the general setting of minimizing the sum of two convex functions. The alternating direction method of multipliers is a form of augmented Lagrangian algorithm that has experienced a renaissance in recent years due to its applicability to optimization problems arising from ``big data'' and image processing applications, and the relative ease with which it may be implemented in parallel and distributed computational environments. There are two fundamental approaches for proving the convergence of the ADMM, each based on a different form of two-way emph{splitting}, that is, expressing a mapping as the sum of two simpler mappings. The first approach is based on Douglas-Rachford operator splitting theory, and yields considerable insight into the convergence of the ADMM. The second convergence proof approach is at its core based on the Lagrangian splitting analysis. We present three new approximate versions of ADMM based on both convergence analyses, all of which require only knowledge of subgradients of the subproblem objectives, rather bounds on the distance to the exact subproblem solution. One version, which applies only to certain common special cases, is based on combining the operator splitting analysis of the ADMM with a relative-error proximal point algorithm of Solodov and Svaiter. A byproduct of this analysis is a new, relative-error version of the Douglas-Rachford splitting algorithm for monotone operators. The other two approximate versions of the ADMM are more general and based on the Lagrangian splitting analysis of the ADMM: one uses a summable absolute error criterion, and the other uses a relative error criterion and an auxiliary iterate sequence. We experimentally compare our new algorithms to an essentially exact form of the ADMM and to an inexact form that can be easily derived from prior theory (but again applies only to certain common special cases). These experiments show that our methods can significantly reduce total computational effort when iterative methods are used to solve ADMM subproblems.

Alternating Direction Method of Multiplier (ADMM) has been a popular algorithmic framework for separable optimization problems with linear constraints. For numerical ADMM fail to exploit the particular structure of the problem at hand nor the input data information, leveraging task-specific modules (e.g., neural networks and other data-driven architectures) to extend ADMM is a significant but challenging task. This work focuses on designing a flexible algorithmic framework to incorporate various task-specific modules (with no additional constraints) to improve the performance of ADMM in real-world applications. Specifically, we propose Guidance from Optimality (GO), a new customization strategy, to embed task-specific modules into ADMM (GO-ADMM). By introducing an optimality-based criterion to guide the propagation, GO-ADMM establishes an updating scheme agnostic to the choice of additional modules. The existing task-specific methods just plug their task-specific modules into the numerical iterations in a straightforward manner. Even with some restrictive constraints on the plug-in modules, they can only obtain some relatively weaker convergence properties for the resulted ADMM iterations. Fortunately, without any restrictions on the embedded modules, we prove the convergence of GO-ADMM regarding objective values and constraint violations, and derive the worst-case convergence rate measured by iteration complexity. Extensive experiments are conducted to verify the theoretical results and demonstrate the efficiency of GO-ADMM.

Distributed Model Predictive Control of linear discrete-time systems with local and global constraints

Data collection plays an important role in wireless sensor networks. Recovery of spatio-temporal data from incomplete sensing data is vital to the network lifetime. Many works have utilized the spatial and temporal correlations to achieve satisfactory data recovery results. However, these methods introduce large computational overhead at the fusion center. In this paper, we develop an ADMM-Net framework for correlated spatio-temporal data recovery. Both the spatial correlation and temporal correlation of sensing data are considered in a convex optimization problem, which is solved by the alternating direction method of multipliers (ADMM) algorithm. We then unfold the ADMM algorithm into a fixed-length neural network that reduces the iterations dramatically and does not require additional location information of nodes. Experimental results on a realworld dataset demonstrate that the proposed method can achieve faster convergence speed than the baseline ADMM algorithm with slight accuracy loss.

Distributed optimization has gained popularity in many areas because it can to cope with the increasing volume of data, and the corresponding demand for computational resources. A popular distributed solver is the alternating direction method of multipliers (ADMM). Fully decentralized (D) ADMM arises when node-to-node communications have to abide by the underlying network connectivity. DADMM’s convergence however, slows down as the network diameter grows large. To deal with this challenge, the recently proposed hybrid (H) ADMM provides considerable performance boost over DADMM by exploiting local topology information. But HADMM only applies to unweighted graphs. The present contribution develops a weighted hybrid (WH) consensus-based ADMM approach that can deal with weighted graphs. The resultant scheme further improves the performance of HADMM through graph-aware weight tuning. Theoretical analysis offers convergence guarantees and establishes linear convergence rate, while numerical tests on various graphs demonstrate the WHADMM merits.

Multi-view clustering utilizes information from diverse views to improve the performance of clustering. For most existing multi-view spectral clustering methods, information of different views is integrated by pursuing a consensus similarity matrix for clustering. However, view-specific structures, which contain the complementary information of multi-view data, may be lost during the clustering process. Actually, in multi-view spectral clustering, similarity matrices of multiple views would have the same clustering structures or properties rather than be numerically uniform. To overcome the aforementioned problem, a novel essential multi-view graph learning (EMGL) method for clustering is proposed in this paper. Different from most existing multi-view spectral clustering, an orthogonal matrix factorization is imposed on multi-view similarity matrices for making them have the same nuclear norm, which indicates the same clustering structures of different views. Furthermore, we also propose an alternating direction method of multipliers (ADMM) based optimization algorithm to address the objective function of our method. Extensive experiments on several datasets demonstrate the superior performance of our proposed method.

The performance of adaptive beamforming suffers from the imprecise knowledge of the array or signal of interest. Many efforts have been made to develop various beamformers and increase their robustness to these unknown errors. This study proposes a new robust adaptive beamformer based on the alternating direction method of multipliers (ADMM). The proposed method is introduced first as an optimisation problem to estimate the steering vector (SV). Owing to its non-convex property, it is converted to a real-valued condition problem by introducing auxiliary variables. Then, the optimisation problem is reformulated as the augmented Lagrangian function. Finally, the estimated SV can be solved by the ADMM. Numerical results prove the effectiveness of the proposed method.

This paper studies the fairness of achievable energy efficiency (EE) in a multicell multiuser multiple-input single-output downlink. The objective is to maximize the minimum EE among all base stations (BSs) subject to per-BS power constraints. The resulting optimization problem is a max-min fractional program, and, thus, difficult to solve in general. Our goal is to develop a decentralized algorithm for the max-min EE problem which solves the problem locally. The idea behind the proposed method is to combine the framework of successive convex approximation (SCA) and alternative direction method of multipliers (ADMM). We transform the convex program obtained at each step of the SCA procedure into a form that lends itself to the ADMM. The resulting formulation is solved optimally by allowing the BSs to exchange the required information until the ADMM converges. In addition to further reduce the backhaul overhead, the proposed algorithm is modified to enhance the convergence speed. Numerical results are provided to demonstrate the effectiveness of the proposed algorithms.

We discuss an online decentralized decision making problem where the agents are coupled with affine inequality constraints. Alternating Direction Method of Multipliers (ADMM) is used as the computation engine and we discuss the convergence of the algorithm in an online setting. To be specific, when decisions have to be made sequentially with a fixed time step, there might not be enough time for the ADMM to converge before the scenario changes and the decision needs to be updated. In this case, a suboptimal solution is employed and we analyze the optimality gap given the convergence condition. Moreover, in many cases, the decision making problem changes gradually over time. We propose a warm-start scheme to accelerate the convergence of ADMM and analyze the benefit of the warm-start. The proposed method is demonstrated in a decentralized multiagent control barrier function problem with simulation.

This paper presents two strategies to speed up the alternating direction method of multipliers (ADMM) for distributed data. In the first method, inspired by stochastic gradient descent, each machine uses only a subset of its data at the first few iterations, speeding up those iterations. A key result is in proving that despite this approximation, our method enjoys the same convergence rate in terms of the number of iterations as the standard ADMM, and hence is faster overall. The second method also follows the idea of sampling a subset of the data to update the model before the communication of each round. It converts an objective to the approximated dual form and performs ADMM on the dual. The method turns out to be a distributed variant of the recently proposed SDCA-ADMM. Yet, compared to the straightforward distributed implementation of SDCA-ADMM, the proposed method enjoys less frequent communication between machines, better memory usage, and lighter computational demand. Experiments demonstrate the effectiveness of our two strategies.

Purpose:To propose a new parallel‐friendly optimization algorithm, based on alternating direction method of multipliers (ADMM), for solving optimization problems emerging in treatment planning and imagingMethods:ADMM, emerging as a powerful tool for distributed optimization, is employed here for IMRT treatment planning. We modify the existing ADMM by integrating that with two optimization techniques known as Barzilai‐Borwein gradient method and line search. We apply original and integrated ADMM, with various penalty parameter values, on three IMRT treatment planning cases and we compare their performance. Each algorithm terminates once it found a solution within the relative error of 1E‐4 from the optimal solution. We also compare the performance of original and integrated ADMM against the commercial optimization software CPLEX.Results:Our experiments on three different cancer sites (prostate, head and neck, and GYN) indicate that integrated ADMM is much faster than original ADMM (5.8 times faster on average). Moreover, while the performance of original ADMM heavily depends on the selected penalty parameter, integrated ADMM performs very robust against the penalty parameter. Compared to the commercial software CPLEX, integrated ADMM turns out to be around 4–7 times faster.Conclusion:Integrated ADMM is a very efficient optimization algorithm to cope with the large‐scale optimization problems. It is a parallel‐friendly optimization algorithm and provides an idea platform for cloud‐based treatment planning and imaging.