In this paper, we study the distributed optimization problem for a system of agents embedded in time-varying directed communication networks. Each agent has its own cost function and agents cooperate to determine the global decision that minimizes the summation of all individual cost functions. We consider the so-called push-pull gradient-based algorithm (termed as AB/Push-Pull) which employs both row- and column-stochastic weights simultaneously to track the optimal decision and the gradient of the global cost while ensuring consensus and optimality. We show that the algorithm converges linearly to the optimal solution over a time-varying directed network for a constant stepsize when the agent's cost function is smooth and strongly convex. The linear convergence of the method has been shown in [F. Saadatniaki, R. Xin, and U.A. Khan, Decentralized optimization over time-varying directed graphs with row and column-stochastic matrices, IEEE Trans. Autom. Control 65(11) (2020), pp. 4769–4780], where the multi-step consensus contraction parameters for row- and column-stochastic mixing matrices are not directly related to the underlying graph structure, and the explicit range for the stepsize value is not provided. With respect to [F. Saadatniaki, R. Xin, and U.A. Khan, Decentralized optimization over time-varying directed graphs with row and column-stochastic matrices, IEEE Trans. Autom. Control 65(11) (2020), pp. 4769–4780], the novelty of this work is twofold: (1) we establish the one-step consensus contraction for both row- and column-stochastic mixing matrices with the contraction parameters given explicitly in terms of the graph diameter and other graph properties; and (2) we provide explicit upper bounds for the stepsize value in terms of the properties of the cost functions, the mixing matrices, and the graph connectivity structure.

In this paper, we investigate accelerated first-order methods for smooth convex optimization problems under inexact information on the gradient of the objective. The noise in the gradient is considered to be additive with two possibilities: absolute noise bounded by a constant, and relative noise proportional to the norm of the gradient. We investigate the accumulation of the errors in the convex and strongly convex settings with the main difference with most of the previous works being that the feasible set can be unbounded. The key to the latter is to prove a bound on the trajectory of the algorithm. We also give a stopping criterion for the algorithm and consider extensions to the cases of stochastic optimization and composite nonsmooth problems.

This paper addresses the worst-case evaluation complexity of a version of the standard quadratic penalty method for smooth nonconvex optimization problems with constraints. The method analysed allows inexact solution of the subproblems and do not require prior knowledge of the Lipschitz constants related with the problem. When an approximate feasible point is used as starting point, it is shown that the referred method takes at most outer iterations to generate an ϵ-approximate KKT point, where is the first penalty parameter. For equality constrained problems, this bound yields to an evaluation complexity bound of , when and suitable first-order methods are used as inner solvers. For problems having only linear equality constraints, an evaluation complexity bound of is established when appropriate p-order methods () are used as inner solvers. Illustrative numerical results are also presented and corroborate the theoretical predictions.

In large-scale optimization, when either forming or storing Hessian matrices are prohibitively expensive, quasi-Newton methods are often used in lieu of Newton's method because they only require first-order information to approximate the true Hessian. Multipoint symmetric secant (MSS) methods can be thought of as generalizations of quasi-Newton methods in that they attempt to impose additional requirements on their approximation of the Hessian. Given an initial Hessian approximation, MSS methods generate a sequence of possibly-indefinite matrices using rank-2 updates to solve nonconvex unconstrained optimization problems. For practical reasons, up to now, the initialization has been a constant multiple of the identity matrix. In this paper, we propose a new limited-memory MSS method for large-scale nonconvex optimization that allows for dense initializations. Numerical results on the CUTEst test problems suggest that the MSS method using a dense initialization outperforms the standard initialization. Numerical results also suggest that this approach is competitive with both a basic L-SR1 trust-region method and an L-PSB method.

The prominent computational features of the Hestenes–Stiefel parameter as one of the fundamental members of conjugate gradient methods have attracted the attention of many researchers. Yet, as a weak stop, it lacks global convergence for general functions. To overcome this defect, a family of spectral version of Hestenes–Stiefel conjugate gradient methods is introduced. To compute the spectral parameter, in the account of worthy properties of quasi-Newton methods, we minimize the distance between the search direction matrix of the spectral conjugate gradient method and the BFGS (Broyden–Fletcher–Goldfarb–Shanno) update. To achieve sufficient descent property, the search direction is projected in the orthogonal subspace to the gradient of the objective function. The convergence analysis of the proposed method is carried out under standard assumptions for general functions. Finally, the practical merits of the suggested method are investigated by numerical experiments on a set of CUTEr test functions using the Dolan–Moré performance profile. The results show the computational efficiency of the proposed method.

It is known [Mangasarian, A Newton method for linear programming, J. Optim. Theory Appl. 121 (2004), pp. 1–18] that every linear program can be solved exactly by minimizing an unconstrained quadratic penalty program. The penalty program is parameterized by a scalar t>0, and one is able to solve the original linear program in this manner when t is selected larger than a finite, but unknown . In this paper, we show that every linear program can be solved using the solution to a parameter-free penalty program. We also characterize the solutions to the quadratic penalty programs using fixed points of certain nonexpansive maps. This leads to an iterative thresholding algorithm that converges to a desired limit point. We show in numerical experiments that this iterative method can outperform a variety of standard quadratic program solvers. Finally, we show that for every , the solution one obtains by solving a parameterized penalty program is guaranteed to lie in the feasible set of the original linear program.

We consider minimizing a class of nonconvex composite stochastic optimization problems, and deterministic optimization problems whose objective function consists of an expectation function (or an average of finitely many smooth functions) and a weakly convex but potentially nonsmooth function. And in this paper, we focus on the theoretical properties of two types of stochastic splitting methods for solving these nonconvex optimization problems: stochastic alternating direction method of multipliers and stochastic proximal gradient descent. In particular, several inexact versions of these two types of splitting methods are studied. At each iteration, the proposed schemes inexactly solve their subproblems by using relative error criteria instead of exogenous and diminishing error rules, which allows our approaches to handle some complex regularized problems in statistics and machine learning. Under mild conditions, we obtain the convergence of the schemes and their computational complexity related to the evaluations on the component gradient of the smooth function, and find that some conclusions of their exact counterparts can be recovered.

In this addendum to our paper published in [Optim. Meth. Softw. 34(1) (2019), pp. 890–920], we provide more details concerning Proposition 2.1 figuring therein. The original proposition implicitly assumed that the constraint mapping was Clarke-regular. Here, not only do we fix some inaccuracies and confusing statements in Section 2 of that paper, but we also provide an extension of our previous results well beyond that setting.

We devise multigrid preconditioners for linear-quadratic space-time distributed parabolic optimal control problems. While our method is rooted in earlier work on elliptic control, the temporal dimension presents new challenges in terms of algorithm design and quality. Our primary focus is on the cG(s)dG(r) discretizations which are based on functions that are continuous in space and discontinuous in time, but our technique is applicable to various other space-time finite element discretizations. We construct and analyse two kinds of multigrid preconditioners: the first is based on full coarsening in space and time, while the second is based on semi-coarsening in space only. Our analysis, in conjunction with numerical experiments, shows that both preconditioners are of optimal order with respect to the discretization in case of cG(1)dG(r) for r = 0, 1 and exhibit a suboptimal behaviour in time for Crank–Nicolson. We also show that, under certain conditions, the preconditioner using full space-time coarsening is more efficient than the one involving semi-coarsening in space, a phenomenon that has not been observed previously. Our numerical results confirm the theoretical findings.

Nowadays, high-dimensional data appear in many practical applications such as biosciences. In the regression analysis literature, the well-known ordinary least-squares estimation may be misleading when the full ranking of the design matrix is missed. As a popular issue, outliers may corrupt normal distribution of the residuals. Thus, since not being sensitive to the outlying data points, robust estimators are frequently applied in confrontation with the issue. Ill-conditioning in high-dimensional data is another common problem in modern regression analysis under which applying the least-squares estimator is hardly possible. So, it is necessary to deal with estimation methods to tackle these problems. As known, a successful approach for high-dimension cases is the penalized scheme with the aim of obtaining a subset of effective explanatory variables that predict the response as the best, while setting the other parameters to zero. Here, we develop several penalized mixed-integer nonlinear programming models to be used in high-dimension regression analysis. The given matrix approximations have simple structures, decreasing computational cost of the models. Moreover, the models are effectively solvable by metaheuristic algorithms. Numerical tests are made to shed light on performance of the proposed methods on simulated and real world high-dimensional data sets.