Computationally relevant generalized derivatives: theory, evaluation and applications

Several software systems are available for implementing automatic differentiation of computer programs. The forward mode of automatic differentiation is limited by computational intensity and computer memory. The reverse mode, or adjoint approach, is limited by computer memory and disk storage. A modular technique for derivative computation that can significantly reduce memory required to compute derivatives in a complex FORTRAN model using the reverse mode of automatic differentiation is discussed and demonstrated.

Recently, non-convex and non-smooth problems have received considerable interests in the fields of image processing and machine learning. The proposed conventional algorithms rely on carefully designed initializations, and the parameters can not be tuned adaptively during iterations with corresponding to various real-world data. To settle these problems, we propose an alternating Bregman network (ABN), which discriminatively learns all the parameters from training pairs and then is directly applied to test data without additional operations. Specifically, parameters of ABN are adaptively learnt from training data to force the objective value drop rapidly toward the optimal and then obtain a desired solution in practice. Furthermore, the basis algorithm of ABN is an alternating method with Bregman modification (AMBM), which solves each subproblem with a designated Bregman distance. This AMBM is more general and flexible than previous approaches; at the same time it is proved to receive the best convergence result for general non-convex and non-smooth optimization problems. Thus, our proposed ABN is an efficient and converged algorithm which rapidly converges to desired solutions in practice. We applied ABN to sparse coding problem with \(\ell _0\) penalty and the experimental results verify the efficiency of our proposed algorithm.

Automatic differentiation (AD) is conventionally understood as a family of distinct algorithms, rooted in two “modes”—forward and reverse—which are typically presented (and implemented) separately. Can there be only one? Following up on the AD systems developed in the JAX and Dex projects, we formalize a decomposition of reverse-mode AD into (i) forward-mode AD followed by (ii) unzipping the linear and non-linear parts and then (iii) transposition of the linear part. To that end, we define a (substructurally) linear type system that can prove a class of functions are (algebraically) linear. Our main results are that forward-mode AD produces such linear functions, and that we can unzip and transpose any such linear function, conserving cost, size, and linearity. Composing these three transformations recovers reverse-mode AD. This decomposition also sheds light on checkpointing, which emerges naturally from a free choice in unzipping let expressions. As a corollary, checkpointing techniques are applicable to general-purpose partial evaluation, not just AD. We hope that our formalization will lead to a deeper understanding of automatic differentiation and that it will simplify implementations, by separating the concerns of differentiation proper from the concerns of gaining efficiency (namely, separating the derivative computation from the act of running it backward).

A modified Polak–Ribière–Polyak conjugate gradient algorithm for nonsmooth convex programs

In this paper, a new class of nonconvex nonsmooth multiobjective programming problems with both inequality and equality constraints is considered. Namely, the sufficient optimality conditions and Mond-Weir duality results are established for such nondifferentiable multicriteria optimization problems in which the involved functions are nondifferentiable (b,?,?,?)w-univex functions (not necessarly with respect to the same functions b and ? and ?). Then the aforesaid results developed here under (b,?,?, ?)w-uinvexity are applicable for a larger class of nonsmooth vector optimization problems than under other generalized convexity notions existing in the literature.

This paper describes an algorithm for solving structured nonsmooth convex optimization problems using the optimal subgradient algorithm (OSGA), which is a first-order method with the complexity mathcal {O}(varepsilon ^{-2}) for Lipschitz continuous nonsmooth problems and mathcal {O}(varepsilon ^{-1/2}) for smooth problems with Lipschitz continuous gradient. If the nonsmoothness of the problem is manifested in a structured way, we reformulate the problem so that it can be solved efficiently by a new setup of OSGA (called OSGA-V) with the complexity mathcal {O}(varepsilon ^{-1/2}). Further, to solve the reformulated problem, we equip OSGA-O with an appropriate prox-function for which the OSGA-O subproblem can be solved either in a closed form or by a simple iterative scheme, which decreases the computational cost of applying the algorithm for large-scale problems. We show that applying the new scheme is feasible for many problems arising in applications. Some numerical results are reported confirming the theoretical foundations.

In non-smooth optimization problems the objective function to minimize or maximize is non-smooth and usually non-convex either, which is a frequent characteristic of real-life optimization problems. In this chapter we discuss the arc routing problem with a non-smooth cost function, and propose a randomized algorithm for solving it. Our approach employs non-uniform probability distributions to add a biased random behavior to the well-known savings heuristic. By doing so, a large set of alternative good solutions can be quickly obtained in a natural way and without complex configuration processes. Since the solution-generation process is based on the criterion of maximizing the savings, it does not need to assume any particular property of the objective function. Therefore, the procedure can be especially useful in problems where properties such as non-smoothness or non-convexity lead to a highly irregulars solution space, for which the traditional optimization methods -both of exact and approximate nature- may fail to reach their full potential. The results obtained so far suggest that using biased probability distributions to randomize classical heuristics can be successfully applied in non-smooth optimization.

Modeling and simulation of phase change and non-ideality in multistream heat exchangers

The subspace technique has been widely used to solve unconstrained/constrained optimization problems and there exist many results that have been obtained. In this paper, a subspace algorithm combining with limited memory BFGS update is proposed for large-scale nonsmooth optimization problems with box-constrained conditions. This algorithm can ensure that all iteration points are feasible and the sequence of objective functions is decreasing. Moreover, rapid changes in the active set are allowed. The global convergence is established under some suitable conditions. Numerical results show that this method is very effective for large-scale nonsmooth box-constrained optimization, where the largest dimension of the test problems is 11,000 variables.

Higher-order KKT optimality conditions through contingent derivatives for constrained nonsmooth vector equilibrium problems

The following overview on non-smooth dynamical systems is given from an application point of view. The aim is to show that real world problems often lead to non-smooth systems and, thus, there is a need to deal with such problems. Moreover, further work shall be stimulated by demonstrating some examples of practical relevance. At present, non-smooth systems are a challenge to engineers and mathematicians as well.

The coupling of different scales in nonlinear systems may lead to some special dynamical phenomena, which always behaves in the combination between large-amplitude oscillations and small-amplitude oscillations, namely bursting oscillations. Up to now, most of therelevant reports have focused on the smooth dynamical systems. However, the coupling of different scales in non-smooth systems may lead to more complicated forms of bursting oscillations because of the existences of different types of non-conventional bifurcations in non-smooth systems. The main purpose of the paper is to explore the coupling effects of multiple scales in non-smooth dynamical systems with non-conventional bifurcations which may occur at the non-smooth boundaries. According to the typical generalized Chua's electrical circuit which contains two non-smooth boundaries, we establish a four-dimensional piecewise-linear dynamical model with different scales in frequency domain. In the model, we introduce a periodically changed current source as well as a capacity for controlling. We select suitable parameter values such that an order gap exists between the exciting frequency and the natural frequency. The state space is divided into several regions in which different types of equilibrium points of the fast sub-system can be observed. By employing the generalized Clarke derivative, different forms of non-smooth bifurcations as well as the conditions are derived when the trajectory passes across the non-smooth boundaries. The case of codimension-1 non-conventional bifurcation is taken for example to investigate the effects of multiple scales on the dynamics of the system. Periodic bursting oscillations can be observed in which codimension-1 bifurcation causes the transitions between the quiescent states and the spiking states. The structure analysis of the attractor points out that the trajectory can be divided into three segments located in different regions. The theoretical period of the movement as well as the amplitudes of the spiking oscillations is derived accordingly, which agrees well with the numerical result. Based on the envelope analysis, the mechanism of the bursting oscillations is presented, which reveals the characteristics of the quiescent states and the repetitive spiking oscillations. Furthermore, unlike the fold bifurcations which may lead to jumping phenomena between two different equilibrium points of the system, the non-smooth fold bifurcation may cause the jumping phenomenon between two equilibrium points located in two regions divided by the non-smooth boundaries. When the trajectory of the system passes across the non-smooth boundaries, non-smooth fold bifurcations may cause the system to tend to different equilibrium points, corresponding to the transitions between quiescent states and spiking states, which may lead to the bursting oscillations.

In this article, a new class of nonconvex nondifferentiable functions, called locally Lipschitz (Φ, ρ)-invex functions, being introduced, includes many well known classes of nondifferentiable generalized convex functions as its subclasses. Some properties of the introduced class of locally Lipschitz (Φ, ρ)-invex functions are studied. Further, nonsmooth mathematical programming problems involving locally Lipschitz (Φ, ρ)-invex functions are considered. Optimality and Mond–Weir duality results for such a class of nonsmooth optimization problems are established.

The paper proposes and develops new globally convergent algorithms of the generalized damped Newton type for solving important classes of nonsmooth optimization problems. These algorithms are based on the theory and calculations of second-order subdifferentials of nonsmooth functions with employing the machinery of second-order variational analysis and generalized differentiation. First we develop a globally superlinearly convergent damped Newton-type algorithm for the class of continuously differentiable functions with Lipschitzian gradients, which are nonsmooth of second order. Then we design such a globally convergent algorithm to solve a structured class of nonsmooth quadratic composite problems with extended-real-valued cost functions, which typically arise in machine learning and statistics. Finally, we present the results of numerical experiments and compare the performance of our main algorithm applied to an important class of Lasso problems with those achieved by other first-order and second-order optimization algorithms.

Bifurcation phenomena in non-smooth dynamical systems