A Nonfeasible Gradient Projection Recurrent Neural Network for Equality-Constrained Optimization Problems

Generalized gradient projection neural network models are proposed to solve nonsmooth convex and nonconvex nonlinear programming problems over a closed convex subset of ℝ n . By using Clarke’s generalized gradient, the neural network modeled by a differential inclusion is developed, and its dynamical behavior and optimization capabilities both for convex and nonconvex problems are rigorously analyzed in the framework of nonsmooth analysis and the differential inclusion theory. First for nonconvex optimization problems, the quasiconvergence results similar to those achieved by previous work are proved. For convex optimization problems, the global convergence results are proved, i.e. any trajectory of the neural network converges to an asymptotically stable equilibrium point, which is an optimal solution of the primal problem whenever it exists. This result remains valid even if the initial condition is chosen outside the feasible set. In addition, an asymptotic control result involving a Tykhonov-like regularization shows that any trajectory of the revised neural network can be forced to converge toward a particular optimal solution of the primal problem. Finally, two simulation optimization algorithms for solving the optimization and control problems are designed respectively, and three typical simulation experiments are illustrated to present the accuracy and efficiency of theoretical convergence results of this paper.

Many engineering applications face the problem of bounding the expected value of a quantity of interest (performance, risk, cost, etc.) that depends on stochastic uncertainties whose probability distribution is not known exactly. Optimal uncertainty quantification (OUQ) is a framework that aims at obtaining the best bound in these situations by explicitly incorporating available information about the distribution. Unfortunately, this often leads to non-convex optimization problems that are numerically expensive to solve. This thesis emphasizes on efficient numerical algorithms for OUQ problems. It begins by investigating several classes of OUQ problems that can be reformulated as convex optimization problems. Conditions on the objective function and information constraints under which a convex formulation exists are presented. Since the size of the optimization problem can become quite large, solutions for scaling up are also discussed. Finally, the capability of analyzing a practical system through such convex formulations is demonstrated by a numerical example of energy storage placement in power grids. When an equivalent convex formulation is unavailable, it is possible to find a convex problem that provides a meaningful bound for the original problem, also known as a convex relaxation. As an example, the thesis investigates the setting used in Hoeffding's inequality. The naive formulation requires solving a collection of non-convex polynomial optimization problems whose number grows doubly exponentially. After structures such as symmetry are exploited, it is shown that both the number and the size of the polynomial optimization problems can be reduced significantly. Each polynomial optimization problem is then bounded by its convex relaxation using sums-of-squares. These bounds are found to be tight in all the numerical examples tested in the thesis and are significantly better than Hoeffding's bounds.

A neurodynamic optimization technique based on overestimator and underestimator functions for solving a class of non-convex optimization problems

The present work explores the stochastic trajectory planning problem considering the parametric and initial condition uncertainties. To address this issue, a novel stochastic trajectory planning (STP) procedure integrating the non-intrusive polynomial chaos expansion and the convex optimization technique is proposed in this paper. This is achieved by transforming the nonconvex and infinite dimensional optimization problem into convex and discrete optimization problem with high efficiency in solving optimal control. Subsequently, the stochastic parametric convex dynamics and path constraints are surrogated by deterministic versions by taking advantage of the high accuracy of the non-intrusive polynomial chaos expansion (NTPCE) without tedious modification of governing equations. The proposed STP procedure is applied to generate the optimal control profile of a three-dimensional interception with the specific impact angle. For comparative studies, the optimal control profile of original optimization problem without disturbance is also derived by the convex optimization technique based on the same scenario. The simulation results show that the optimal control profile obtained by the proposed STP approach is more robust to the parametric and initial condition uncertainties under the open-loop control policy and don’t need significant increase of computational cost. These results demonstrate the effectiveness of the proposed procedure.

Waveform-agile tracking performance of cognitive tracking radar is significantly affected by the target maneuverability and increasing radio frequency interference (RFI) of other in-band systems. Therefore, a joint spectrum management and waveform optimization (JSMWO) policy is proposed to improve the maneuvering target tracking performance while mitigating RFI. First, the JSMWO policy is mathematically formulated to simultaneously optimize the spectrum occupancy and waveform parameters by minimizing a JSMWO-oriented three-term objective function. Specifically, the first two terms consider the tradeoff between interference avoidance and bandwidth utilization, and the impact of current spectrum occupancy action on future multistep rewards. The last subreward term incorporates the criterion-specific conditional predictive Bayesian risk, waveform library limitation, and real-time spectrum constraint generated by spectrum occupancy action. Second, to resolve the underlying mixed-integer, nonlinear, nonconvex multivariable optimization problem, by proper coupling analysis, we develop a four-stage iterative method to decompose the original JSMWO policy into two subpolicies and tackle them sequentially, which encompasses the deep Q-learning (DQL) based spectrum occupancy optimization, particle swarm optimization (PSO) based waveform optimization, and feedback information-based online learning. Specifically, the dual-network architecture and experience replay mechanism equipped in DQL can effectively address the delay reward and future multistep effects during the spectrum occupancy subpolicy optimization. Waveform optimization subpolicy is subsequently reformulated as an equivalent constrained optimization problem, wherein the PSO can effectively optimize the waveform parameters through interparticle interactions and iterations while satisfying the spectrum constraint and waveform limitation. Simulation results demonstrate the effectiveness and superiority of the above-mentioned JSMWO policy.

A solution algorithm for non-convex mixed integer optimization problems with only few continuous variables

In recent years, research is very active in nonconvex optimization. There are two principal reasons for this: The first is the importance of its applications to concrete problems in practice. The second is a natural way of leaving the convex optimization (which is sufficiently studied and can be considered as practically solved) and passing to the nonconvex optimization. More especially as the resolution of a nonconvex optimization problem requires in general, at each step, the resolution of a convex optimization problem; and then it is necessary to adapt and to make efficient the existent algorithms of convex optimization for solving the nonconvex optimization problems.

This paper investigates the linear precoder optimization for packet retransmissions in nonregenerative multi-input multi-output (MIMO) relay systems. To fully utilize the time diversity provided by automatic repeat request (ARQ), progressive linear precoders are designed with the objective of maximizing the mutual information delivered by multiple transmissions of the same packet. Firstly, the relay-only precoder is considered when the channel state information (CSI) is only available at the relay and the destination. We derive the form of the optimal ARQ precoder by restating the optimization problem as a matrix diagonalization problem, then the power allocation is formulated as a convex optimization problem where the Karush-Kuhn-Tucker conditions are used to obtain the optimal solution. Furthermore, we analyze the asymptotic performance when the two hops' signal-to-noise ratio (SNR) goes to infinity or zero respectively. Secondly, the joint source and relay precoder is studied assuming the CSI known at all nodes. The ARQ precoder is derived by matrix diagonalization and channel pairing between previous subchannels and current subchannels. Then the power allocation at the source and relay is formulated as a nonconvex multi-variable optimization problem where we employ an iterative algorithm to obtain the suboptimal solution. Moreover, the asymptotic performance is also analyzed with the two hops' SNR being extremely large or small, based on which the simplified method of searching channel pairing matrix is proposed. Numerical results show that the proposed precoders can improve the system performance significantly, especially in the joint source and relay precoder scenario.

In this paper, we present a new nonmonotone memory gradient algorithm for unconstrained optimization problems. An attractive property of the proposed method is that the search direction always provides sufficient descent step at each iteration. This property is independent of the line search used. Under mild assumptions, the global and local convergence results of the proposed algorithm are established respectively. Numerical results are also reported to show that the proposed method is suitable to solve large-scale optimization problems and is more stable than other similar methods in practical computation.

We consider fusing an arbitrary number of multiband, i.e., panchromatic, multispectral, or hyperspectral, images of the same scene. Using the well-known forward observation and linear mixture models, a vector total-variation penalty, and appropriate constraints, we cast this problem as a convex optimization problem. The total-variation penalty helps cope with the potential ill-posedness of the underlying inverse problem by exploiting the prior knowledge that natural images are mostly piecewise smooth in the spatial domain and comprise relatively few archetypical signatures, i.e., endmembers, in the spectral domain. We solve the formulated convex but non-smooth optimization problem using the alternating direction method of multipliers. Our experiments with multiband images constructed from real hyperspectral datasets demonstrate the performance advantages of the proposed algorithm over the existing algorithms, which need to be used in tandem to fuse more than two multiband images.

An exponential penalty function (EPF) formulation based on method of multipliers is presented for solving multilevel optimization problems within the framework of analytical target cascading. The original all-at-once constrained optimization problem is decomposed into a hierarchical system with consistency constraints enforcing the target-response coupling in the connected elements. The objective function is combined with the consistency constraints in each element to formulate an augmented Lagrangian with EPF. The EPF formulation is implemented using double-loop (EPF I) and single-loop (EPF II) coordination strategies and two penalty-parameter-updating schemes. Four benchmark problems representing nonlinear convex and non-convex optimization problems with different number of design variables and design constraints are used to evaluate the computational characteristics of the proposed approaches. The same problems are also solved using four other approaches suggested in the literature, and the overall computational efficiency characteristics are compared and discussed.

In this paper, the classical exact absolute value penalty function method is used for solving a new class of nonconvex nonsmooth optimization problems. Nonconvex nondifferentiable optimization problems with both inequality and equality constraints are considered here, in which not all functions constituting them have the fundamental property of convex functions and most classes of generalized convex functions—namely, a stationary point of such a function is its global minimum. It is proved for such nonconvex optimization problems that there exists a finite threshold of penalty parameters equal to the largest absolute value of a Lagrange multiplier such that, for every penalty parameter exceeding this lower bound, there is the equivalence between an optimal solution in the original constrained minimization problem with ‐invex functions and a minimizer in its associated penalized optimization problem with the exact l1 penalty function. Further, under nondifferentiable ‐invexity assumptions, a characterization of a saddle point of the Lagrange function, defined for the considered constrained optimization problem in terms of minimizers of its associated exact penalized optimization problem with the exact l1 penalty function, is presented.

Recently crowdsourcing services are often used to collect a large amount of labeled data for machine learning, since they provide us an easy way to get labels at very low cost and in a short period. The use of crowdsourcing has introduced a new challenge in machine learning, that is, coping with the variable quality of crowd-generated data. Although there have been many recent attempts to address the quality problem of multiple workers, only a few of the existing methods consider the problem of learning classifiers directly from such noisy data. All these methods modeled the true labels as latent variables, which resulted in non-convex optimization problems. In this paper, we propose a convex optimization formulation for learning from crowds without estimating the true labels by introducing personal models of the individual crowd workers. We also devise an efficient iterative method for solving the convex optimization problems by exploiting conditional independence structures in multiple classifiers. We evaluate the proposed method against three competing methods on synthetic data sets and a real crowdsourced data set and demonstrate that the proposed method outperforms the other three methods.

As for deployment of security strategy, corresponding forwarding rules for switches can be given in allusion to different traffic conditions. However, due to lack of global cognitive control for security strategy deployment in traditional Internet of Vehicles (IoV), it is quite difficult to realize global and optimized security strategy deployment scheme so as to meet security requirements in different traffic conditions. On basis of traditional IoV, cognitive engine is added in cognitive IoV (CIoV) to enhance the intelligence of traditional IoV. In allusion to CIoV, and in consideration of restrictions on transmission delay, the security strategy deployment for switches on core network is formulated in this paper, thus not only the safe transmission rules are met, but the transmission delay can also be the lowest. To be specific, the path selection of switches is modeled as 0-1 programming problem in this paper, and that optimization problem is proved to be a nonconvex optimization problem. Then we convert that problem into a convex optimization problem by log-det heuristic algorithm, thus to give path selection scheme to meet security requirements with the lowest delay on the whole. Experiment proves that cognitive engine-based security strategy deployment put forth in this paper is much better than other schemes.

In principle component pursuit (PCP), the essential idea is to replace the original non-convex optimization problem of the matrix rank and the count of non-zero entries by a convex optimization problem of the nuclear and I <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norms. In the PCP literature, it is rigorously proved that the validity of this idea depends on the coherence of the uncontaminated data. Specifically, the lower the coherence is, the equivalence of the convex optimization problem to the original non-convex one will hold by a larger probability. Although the coherence index is fixed for a given data set, it is possible to adjust this index by introducing different scalings to the data. The target of this work is thus to find the optimal scaling of the data such that the coherence index is minimized. Based on the analysis of the PCP problem structure, a non-convex optimization problem with implicit dependence on the scaling parameters is firstly formulated. To solve this problem, a coordinate descent algorithm is proposed. Under mild conditions on the structure of the data matrix, the convergence of the algorithm to a global optimal point is rigorously proved by treating the algorithm as a discrete-time dynamic system and utilizing a Lyapunov-type approach. Monte Carlo simulation experiments are performed to verify the effectiveness of the developed results.