Quadratic Support Functions in Quadratic Bilevel Problems

This paper addresses a bilevel optimization problem with a guaranteed (pessimistic) solution in a rather general form. The problem is reduced to a family of bilevel problems with optimistic solutions and perturbed follower's problems. For the quadratic-linear bilevel optimization problem, a reduction of the bilevel problem to a series of non-convex one-level optimization problems is proposed using the penalty method. Additionally, a solution method based on Global Search Theory (GST) is developed for the obtained problem. The GST consists of two stages: special local search and Global Search Scheme (GSS), which is based on Global Optimality Conditions (GOCs). The computational testing of the developed approach on specially constructed bilevel problems of high dimensions was successfully carried out. Finally, the computational testing on specially constructed bilevel problems of rather high dimensions was successfully carried out, that substantiate the developed approach.

Presolving linear bilevel optimization problems

Due to the first chapter we realize that convex programs of the type (5.4) with random parameters (η,ξ) in the objective and in the right-hand side play an essential role in stochastic two-stage programs. We are aware of the fact that we have to integrate (at least approximately) the optimal value function (recourse function) of (5.4), given — for a fixed first-stage decision x — only implicitly by the infimum of (5.4). Although the optimal value function changes with x, we take into account that it remains a saddle function. This fact motivates to derive suitable approximates for the optimal value function, that are computable and allow conclusions about the goodness of the first-stage decision x. Such approximates will be derived in chapter III. At this point it is only of importance to stress that the existence of these approximates essentially depend on the existence of subgradients of the optimal value function at distinguished points. Hence, at first, we are confronted with investigating under which assumptions subgradients for the optimal value function at a certain distinguished point, say (η0,ξ0),exist. In other words we have to ensure dual solvability of the convex optimization problem — of the type (5.4) — with respect to (η,ξ) at (η0,ξ0).

Bilevel optimization problems have received a lot of attention in the last years and decades. Besides numerous theoretical developments there also evolved novel solution algorithms for mixed-integer linear bilevel problems and the most recent algorithms use branch-and-cut techniques from mixed-integer programming that are especially tailored for the bilevel context. In this paper, we consider MIQP-QP bilevel problems, i.e., models with a mixed-integer convex-quadratic upper level and a continuous convex-quadratic lower level. This setting allows for a strong-duality-based transformation of the lower level which yields, in general, an equivalent nonconvex single-level reformulation of the original bilevel problem. Under reasonable assumptions, we can derive both a multi- and a single-tree outer-approximation-based cutting-plane algorithm. We show finite termination and correctness of both methods and present extensive numerical results that illustrate the applicability of the approaches. It turns out that the proposed methods are capable of solving bilevel instances with several thousand variables and constraints and significantly outperform classical solution approaches.

Purpose: To support IMRT research, we developed an open‐source, computationally efficient, and flexible optimization solver that can solve a wide range of problem formulations, including constrained optimization, convex and non‐convex polynomial functions, and hierarchical optimization formulations. Methods: The solver is an adaptation and modification of the open‐source Interior Point Optimization Solver (IPOPT), with some performance enhancements. Previously, we implemented the IMRT optimization based on the commercial Mosek solver. Mosek can solve convex linear and quadratic problems but cannot solve convex or non‐convex polynomials of higher orders. This limits the ability to model IMRT optimization using more appropriate metrics like gEUD. Therefore, we implemented the optimization problem using C++ interface of IPOPT. IPOPT can use the Newton or Quasi‐Newton method to solve the optimization problem. In the Newton method, the exact Hessian matrix provided by the user is used to find the direction in each iteration while in the Quasi‐Newton method the Hessian matrix is approximated from the Jacobian matrix. After integrating IPOPT with CERR, we found that the Newton method converges more quickly than the Quasi‐Newton method. To increase the computational efficiency, we incorporated the uBLAS matrix library. Results: Few compared the modified IPOPT code to Mosek for a set of convex quadratic problems using realistic, anonymized treatment planning data and dose data accessed using CERR. These problems have a unique global minimum and both solvers were observed to give the same results for eight head and neck planning datasets. The median speedup of IPOPT compared to Mosek was a factor of 3.5. Conclusion: Our IPOPT implementation provides an open‐source framework to model the IMRT optimization using complex objective functions; is integrated with CERR; and our C++ implementation of the optimization problem gives increased control over the optimization process. This tool provides a basis for future IMRT treatment planning research. Memorial Sloan‐Kettering Cancer Center

Optimal dividend policies for compound Poisson processes: The case of bounded dividend rates

In Reinforcement Learning, an intelligent agent has to make a sequence of decisions to accomplish a goal. If this sequence is long, then the agent has to plan over a long horizon. While learning the optimal policy and its value function is a well studied problem in Reinforcement Learning, this paper focuses on the structure of the optimal value function and how hard it is to represent the optimal value function. We show that the generalized Rademacher complexity of the hypothesis space of all optimal value functions is dependent on the planning horizon and independent of the state and action space size. Further, we present bounds on the action-gaps of action value functions and show that they can collapse if a long planning horizon is used. The theoretical results are verified empirically on randomly generated MDPs and on a grid-world fruit collection task using deep value function approximation. Our theoretical results highlight a connection between value function approximation and the Options framework and suggest that value functions should be decomposed along bottlenecks of the MDP's transition dynamics.

Linear bilevel programs with multiple objectives at the upper level

The problem of finding the best rank-one approximation to higher-order tensors has extensive engineering and statistical applications. It is well-known that this problem is equivalent to a homogeneous polynomial optimization problem. In this paper, we study theoretical results and numerical methods of this problem, particularly focusing on the 4-th order symmetric tensor case. First, we reformulate the polynomial optimization problem to a matrix programming, and show the equivalence between these two problems. Then, we prove that there is no duality gap between the reformulation and its Lagrangian dual problem. Concerning the approaches to deal with the problem, we propose two relaxed models. The first one is a convex quadratic matrix optimization problem regularized by the nuclear norm, while the second one is a quadratic matrix programming regularized by a truncated nuclear norm, which is a D.C. function and therefore is nonconvex. To overcome the difficulty of solving this nonconvex problem, we approximate the nonconvex penalty by a convex term. We propose to use the proximal augmented Lagrangian method to solve these two relaxed models. In order to obtain a global solution, we propose an alternating least eigenvalue method after solving the relaxed models and prove its convergence. Numerical results presented in the last demonstrate, especially for nonpositive tensors, the effectiveness and efficiency of our proposed methods.

In this paper, we adopt an iterative approach to solve the class of optimization problem for the sum of finite functions over split equality optimization problems for the sum of two functions. This type of problem contains many optimization problems, and bilevel problems, as well as split equality problems, and split feasibility problems as special cases. Here, we are able to establish a strong convergence theorem for an iterative method for solving this problem. As consequences of this convergence theorem, we study the following problems: optimization for the sum of finite functions over the common solution set of optimization problems for the sum of two functions; optimization for the sum of finite functions; optimization for the sum of finite functions with split equality inconsistent feasibility constraints; optimization for the sum of finite functions over the solution set for split equality constrained quadratic signal recovery problem; optimization for the sum of finite functions over the solution set of generalized split equality multiple set feasibility problem, and optimization for the sum of finite functions over the solution set of split equality linear equations problem. We use simultaneous iteration to establish strong convergence theorems for these problems. Our results generalize and improve many existing theorems for these types of problems in the literature and will have applications in nonlinear analysis, optimization problems and signal processing problems.

We describe an approximate dynamic programming method for stochastic control problems on infinite state and input spaces. The optimal value function is approximated by a linear combination of basis functions with coefficients as decision variables. By relaxing the Bellman equation to an inequality, one obtains a linear program in the basis coefficients with an infinite set of constraints. We show that a recently introduced method, which obtains convex quadratic value function approximations, can be extended to higher order polynomial approximations via sum of squares programming techniques. An approximate value function can then be computed offline by solving a semidefinite program, without having to sample the infinite constraint. The policy is evaluated online by solving a polynomial optimization problem, which also turns out to be convex in some cases. We experimentally validate the method on an autonomous helicopter testbed using a 10-dimensional helicopter model.

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

Prioritized sweeping (PS) and its variants are model-based reinforcement-learning algorithms that have demonstrated superior performance in terms of computational and experience e‐ciency in practice. This note establishes the flrst|to the best of our knowledge|formal proof of convergence to the optimal value function when they are used as planning algorithms. We also describe applications of this result to provably e‐cient model-based reinforcement learning in the PAC-MDP framework. We do not address the issue of convergence rate in the present paper.

Bilevel problems are highly challenging optimization problems that appear in many applications of energy market design, critical infrastructure defense, transportation, pricing, and so on. Often these bilevel models are equipped with integer decisions, which makes the problems even harder to solve. Typically, in such a setting in mathematical optimization, one develops primal heuristics in order to obtain feasible points of good quality quickly or to enhance the search process of exact global methods. However, there are comparably few heuristics for bilevel problems. In this paper, we develop such a primal heuristic for bilevel problems with a mixed-integer linear or quadratic upper level and a linear or quadratic lower level. The heuristic is based on a penalty alternating direction method, which allows for a theoretical analysis. We derive a convergence theory stating that the method converges to a stationary point of an equivalent single-level reformulation of the bilevel problem and extensively test the method on a test set of more than 2,800 instances—which is one of the largest computational test sets ever used in bilevel programming. The study illustrates the very good performance of the proposed method in terms of both running times and solution quality. This renders the method a suitable subroutine in global bilevel solvers as well as a reasonable standalone approach.Summary of Contribution: Bilevel optimization problems form a very important class of optimization problems in the field of operations research, which is mainly due to their capability of modeling hierarchical decision processes. However, real-world bilevel problems are usually very hard to solve—especially in the case in which additional mixed-integer aspects are included in the modeling. Hence, the development of fast and reliable primal heuristics for this class of problems is very important. This paper presents such a method.

A theoretical framework for the reinforcement learning of aesthetic biases was recently proposed based on brain circuitries revealed by neuroimaging. A model grounded on that framework accounted for interesting features of human aesthetic biases. These features included individuality, cultural predispositions, stochastic dynamics of learning and aesthetic biases, and the peak-shift effect. However, despite the success in explaining these features, a potential weakness was the linearity of the value function used to predict reward. This linearity meant that the learning process employed a value function that assumed a linear relationship between reward and sensory stimuli. Linearity is common in reinforcement learning in neuroscience. However, linearity can be problematic because neural mechanisms and the dependence of reward on sensory stimuli were typically nonlinear. Here, we analyze the learning performance with models including optimal nonlinear value functions. We also compare updating the free parameters of the value functions with the delta rule, which neuroscience models use frequently, vs. updating with a new Phi rule that considers the structure of the nonlinearities. Our computer simulations showed that optimal nonlinear value functions resulted in improvements of learning errors when the reward models were nonlinear. Similarly, the new Phi rule led to improvements in these errors. These improvements were accompanied by the straightening of the trajectories of the vector of free parameters in its phase space. This straightening meant that the process became more efficient in learning the prediction of reward. Surprisingly, however, this improved efficiency had a complex relationship with the rate of learning. Finally, the stochasticity arising from the probabilistic sampling of sensory stimuli, rewards, and motivations helped the learning process narrow the range of free parameters to nearly optimal outcomes. Therefore, we suggest that value functions and update rules optimized for social and ecological constraints are ideal for learning aesthetic biases.