Stochastic proximal linear method for structured non-convex problems

This paper deals with composite optimization problems having the objective function formed as the sum of two terms; one has a Lipschitz continuous gradient along random subspaces and may be nonconvex, and the second term is simple and differentiable but possibly nonconvex and nonseparable. Under these settings, we design a stochastic coordinate proximal gradient method that takes into account the nonseparable composite form of the objective function. This algorithm achieves scalability by constructing at each iteration a local approximation model of the whole nonseparable objective function along a random subspace with user-determined dimension. We outline efficient techniques for selecting the random subspace, yielding an implementation that has low cost per iteration, also achieving fast convergence rates. We present a probabilistic worst case complexity analysis for our stochastic coordinate proximal gradient method in convex and nonconvex settings; in particular, we prove high-probability bounds on the number of iterations before a given optimality is achieved. Extensive numerical results also confirm the efficiency of our algorithm. Funding: This work was supported by Norway Grants 2014-2021 [Grant ELO-Hyp 24/2020]; Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii [Grants PN-III-P4-PCE-2021-0720, L2O-MOC, nr 70/2022]; and the ITN-ETN project TraDE-OPT funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement [Grant 861137].

In this paper, we consider a class of structured nonconvex nonsmooth optimization problems, in which the objective function is formed by the sum of a possibly nonsmooth nonconvex function and a differentiable function with Lipschitz continuous gradient, subtracted by a weakly convex function. This general framework allows us to tackle problems involving nonconvex loss functions and problems with specific nonconvex constraints, and it has many applications such as signal recovery, compressed sensing, and optimal power flow distribution. We develop a proximal subgradient algorithm with extrapolation for solving these problems with guaranteed subsequential convergence to a stationary point. The convergence of the whole sequence generated by our algorithm is also established under the widely used Kurdyka–Łojasiewicz property. To illustrate the promising numerical performance of the proposed algorithm, we conduct numerical experiments on two important nonconvex models. These include a compressed sensing problem with a nonconvex regularization and an optimal power flow problem with distributed energy resources.

In this paper, we propose a redistributed proximal bundle method for a class of nonconvex nonsmooth optimization problems with inexact information, i.e., we consider the problem of computing the approximate critical points when only the inexact information about the function values and subgradients are available and show that reasonable convergence properties are obtained. We assume that the errors in the computation of functions and subgradients are only bounded and in principle do not have to vanish within the limits. For the nonconvex functions, we design the convexification technique, which ensures that the linearization error of its augmentation function is nonnegative. Meanwhile, for the inexact information, we utilize noise management strategies and update approximate parameters to reduce the impact of inexact information. Based on this method, we can obtain the approximate solution.

In this article, we investigate the possibilities of accelerating the double smoothing (DS) technique when solving unconstrained nondifferentiable convex optimization problems. This approach relies on the regularization in two steps of the Fenchel dual problem associated with the problem to be solved into an optimization problem having a differentiable strongly convex objective function with Lipschitz continuous gradient. The doubly regularized dual problem is then solved via a fast gradient method. The aim of this article is to show how the properties of the functions in the objective of the primal problem influence the implementation of the DS approach and its rate of convergence. The theoretical results are applied to linear inverse problems by making use of different regularization functionals.

In a separable real Hilbert space, we study the problem of minimizing a convex function with Lipschitz continuous gradient in the presence of noisy evaluations. To this end, we associate a stochastic Heavy Ball system, incorporating a friction coefficient, with the optimization problem. We establish existence and uniqueness of trajectory solutions for this system. Under a square integrability condition for the diffusion term, we prove almost sure convergence of the trajectory process to an optimal solution, as well as almost sure convergence of its time derivative to zero. Moreover, we derive almost sure and expected convergence rates for the function values along the trajectory towards the infimal value. Finally, we show that the stochastic Heavy Ball system is equivalent to a Su-Boyd-Candès-type system for a suitable choice of the parameter function, and we provide corresponding convergence rate results for the latter.In the second part of this paper, we extend our analysis beyond the optimization framework and investigate a monotone equation induced by a monotone and Lipschitz continuous operator, whose evaluations are assumed to be corrupted by noise. As before, we consider a stochastic Heavy Ball system with a friction coefficient and a correction term, now augmented by an additional component that accounts for the time derivative of the operator. We establish analogous convergence results for both the trajectory process and its time derivative, and derive almost sure as well as expected convergence rates for the decay of the residual and the gap function along the trajectory. As a final result, we show that a particular instance of the stochastic Heavy Ball system for monotone equations is equivalent to a stochastic second-order dynamical system with a vanishing damping term. Remarkably, this system exhibits fast convergence rates for both the residual and gap functions.

This paper proposes a novel nonmonotone proximal gradient method for solving nondifferentiable composite multiobjective optimization problems. The method integrates a nonmonotone line search technique into the proximal gradient framework, offering enhanced flexibility and efficiency for handling nonsmooth or nonconvex objective functions. We analyze the convergence properties of the proposed algorithm, demonstrate that all accumulation points of the generated sequence satisfy Pareto stationarity when the problem is nonconvex. Furthermore, under convexity and mild conditions, we establish that the sequence converges to a weakly Pareto optimal solution of the problem. Numerical experiments on robust multiobjective optimization problems and Lasso multiobjective optimization problems are provided to illustrate the computational efficiency and the practical utility of the proposed method.

Many problems arising in image processing and signal recovery with multi-regularization and constraints can be formulated as minimization of a sum of three convex separable functions. Typically, the objective function involves a smooth function with Lipschitz continuous gradient, a linear composite nonsmooth function, and a nonsmooth function. In this paper, we propose a primal-dual fixed point (PDFP) scheme to solve the above class of problems. The proposed algorithm for three-block problems is a symmetric and fully splitting scheme, only involving an explicit gradient, a linear transform, and the proximity operators which may have a closed-form solution. We study the convergence of the proposed algorithm and illustrate its efficiency through examples on fused LASSO and image restoration with non-negative constraint and sparse regularization.

This paper analyzes block-coordinate proximal gradient methods for minimizing the sum of a separable smooth function and a (nonseparable) nonsmooth function, both of which are allowed to be nonconvex. The main tool in our analysis is the forward-backward envelope, which serves as a particularly suitable continuous and real-valued Lyapunov function. Global and linear convergence results are established when the cost function satisfies the Kurdyka–Łojasiewicz property without imposing convexity requirements on the smooth function. Two prominent special cases of the investigated setting are regularized finite sum minimization and the sharing problem; in particular, an immediate byproduct of our analysis leads to novel convergence results and rates for the popular Finito/MISO algorithm in the nonsmooth and nonconvex setting with very general sampling strategies.

Proximal gradient method for convex multiobjective optimization problems without Lipschitz continuous gradients

We introduce and analyze BPALM and A-BPALM, two multi-block proximal alternating linearized minimization algorithms using Bregman distances for solving structured nonconvex problems. The objective function is the sum of a multi-block relatively smooth function (i.e., relatively smooth by fixing all the blocks except one) and block separable (nonsmooth) nonconvex functions. The sequences generated by our algorithms are subsequentially convergent to critical points of the objective function, while they are globally convergent under the KL inequality assumption. Moreover, the rate of convergence is further analyzed for functions satisfying the Łojasiewicz’s gradient inequality. We apply this framework to orthogonal nonnegative matrix factorization (ONMF) that satisfies all of our assumptions and the related subproblems are solved in closed forms, where some preliminary numerical results are reported.

Optimal design of hysteretic dampers connecting adjacent structures using multi-objective genetic algorithm and stochastic linearization method

Unmixing of hyperspectral data is an area of major research because the information it provides is utilized in plethora of fields. The year of 2006 witnessed the emergence of Compressed Sensing algorithm which was later used to spearhead research in umixing problems. Later, the notion of \(\ell _p\) norms \(0< p < 1\) and other non-smooth and non-convex penalty function were used in place of the traditional convex \(\ell _1\) penalty. Dealing with optimization problems with non-convex objective function is rather difficult as most methodologies often get stuck at local optima. In this paper, a parameterised non-convex penalty function is used to induce sparsity in the unknown.The parameters of penalty function can be adjusted so as to make the objective function convex, thus resulting in the possibility of finding a global optimal solution. Here ADMM algorithm is utilized to arrive at the final iterative algorithm for the unmixing problem. The algorithm is tested on synthetic data set, generated from the spectral library provided by US geological survey. Different parametric penalty functions like \(\log \) and \(\arctan \) are used in the algorithm and is compared with the traditional \(\ell _1\) penalties, in terms of the performance measures RSNR and PoS. It was observed that the non-convex penalty functions out-performs the \(\ell _1\) penalty in terms of the aforementioned measures.

This paper focuses on the minimization of a sum of a twice continuously differentiable function f and a nonsmooth convex function. An inexact regularized proximal Newton method is proposed by an approximation to the Hessian of f involving the ϱ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho $$\\end{document}th power of the KKT residual. For ϱ=0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho =0$$\\end{document}, we justify the global convergence of the iterate sequence for the KL objective function and its R-linear convergence rate for the KL objective function of exponent 1/2. For ϱ∈(0,1)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho \\in (0,1)$$\\end{document}, by assuming that cluster points satisfy a locally Hölderian error bound of order q on a second-order stationary point set and a local error bound of order q>1+ϱ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$q>1\\!+\\!\\varrho $$\\end{document} on the common stationary point set, respectively, we establish the global convergence of the iterate sequence and its superlinear convergence rate with order depending on q and ϱ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varrho $$\\end{document}. A dual semismooth Newton augmented Lagrangian method is also developed for seeking an inexact minimizer of subproblems. Numerical comparisons with two state-of-the-art methods on ℓ1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _1$$\\end{document}-regularized Student’s t-regressions, group penalized Student’s t-regressions, and nonconvex image restoration confirm the efficiency of the proposed method.

This paper focuses on the large-scale composite optimization problem, which is the sum of a finite number of smooth (possibly nonconvex) functions and a convex (possibly nonsmooth) function. A common method to solve this problem is proximal incremental aggregated gradient (PIAG) method (N. D. Vanli, M. Gurbuzbalaban and A. Ozdaglar, SIAM Journal on Optimization, 2018, 28(2): 1282–1300), which exploits the gradient information from previous iterations to approximate the gradient at the current point. However, as many first-order algorithms, it may suffer from slow convergence. To address this issue, this paper proposes an inexact variable metric variant of incremental aggregated Forward-Backward (iVMFB-IAG) method. This methods incorporates a variable metric strategy to accelerate the PIAG method and introduces an inexact criterion to enhance its practicality. Under the Kurdyka-Łojasiewicz (KL) property and the uniformly bounded positive definiteness of the scaling matrix, the sequence generated by the iVMFB-IAG method converges globally to the critical point of the problem. Additionally, the convergence rate is established under the KL framework. Moreover, under certain restrictive conditions, the method exhibits local superlinear convergence. Through the numerical experiments on image reconstruction applications, we demonstrate the effectiveness of the iVMFB-IAG method by restoring two distinct images.

In this paper, we explore a specific optimization problem that involves the combination of a differentiable nonconvex function and a nondifferentiable function. The differentiable component lacks a global Lipschitz continuous gradient, posing challenges for optimization. To address this issue and accelerate the convergence, we propose a Bregman proximal stochastic gradient method with extrapolation (BPSGE), which only requires smooth adaptivity of the differentiable part. Under variance reduction framework, we not only analyze the subsequential and global convergence of the proposed algorithm under certain conditions, but also analyze the sublinear convergence rate of the subsequence, and the complexity of the algorithm, revealing that the BPSGE algorithm requires at most O(epsilon\^\,(-2)) iterations in expectation to attain an epsilon-stationary point. To validate the effectiveness of our proposed algorithm, we conduct numerical experiments on three real-world applications: graph regularized nonnegative matrix factorization (NMF), matrix factorization with weakly-convex regularization, and NMF with nonconvex sparsity constraints. These experiments demonstrate that BPSGE is faster than the baselines without extrapolation. The code is available at: https://github.com/nothing2wang/BPSGE-Algorithm.