A randomisation method for mean-field control problems with common noise

This chapter deals with completely observable stochastic control problems for diffusion processes, described by SDEs. The decision maker chooses an optimal decision among all possible ones to achieve the goal. Namely, for a control process, its response evolves according to a (controlled) SDE and the payoff on a finite time interval is given. The controller wants to minimize (or maximize) the payoff by choosing an appropriate control process from among all possible ones. Here we consider three types of control processes: 1. \((\mathcal{F}_{t})\)-progressively measurable processes. 2. Brownian-adapted processes. 3. Feedback controls. In order to analyze the problems, we mainly use the dynamic programming principle (DPP) for the value function.The reminder of this chapter is organized as follows. Section 2.1 presents the formulation of control problems and basic properties of value functions, as preliminaries for later sections. Section 2.2 focuses on DPP. Although DPP is known as a two stage optimization method, we will formulate DPP by using a semigroup and characterize the value function via the semigroup. In Sect. 2.3, we deal with verification theorems, which give recipes for finding optimal Markovian policies. Section 2.4 considers a class of Merton-type optimal investment models, as an application of previous results.KeywordsOptimal Markovian PoliciesDynamic Programming Principle (DPP)Verification TheoremStochastic Control ProblemStage Optimization MethodThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Infinite horizon singular optimal control problems with control taking values in a closed cone U in Rn lead to a dynamic programming equation of the form: maxF2x,v, v',v'' ,F1x,v,v' =0, forall x∈Q, where Q , the state space of the control problem, is some nonempty connected open subset of Rn , and F1, F2 are continuous real-valued functions on QxR2 and QxR3 respectively, with the coercivity assumption for F 2, that is, the function r&rarrr;F2x,u,p,r is nonincreasing on R . A major concern is to determine how smooth the value function v is across the free boundary of the problem. Linear-convex deterministic and stochastic singular control problems in dimension one are considered. We present the analysis of the above equation together with smoothness of the value function across the free boundary. The interest here is the explicitness of the results and the fact that the smooth fit principle depends on the parameters of the control problem. In particular, the value function for the stochastic control problem is founded explicitly, and at the same time, optimal controls are identified using a verification theorem.

Constrained BSDEs representation of the value function in optimal control of pure jump Markov processes

This article is concerned with stochastic control problems for backward doubly stochastic differential equations of mean-field type, where the coefficient functions depend on the joint distribution of the state process and the control process. We obtain the stochastic maximum principle which serves as a necessary condition for an optimal control, and we also prove its sufficiency under proper conditions. As a byproduct, we prove the well-posedness for a type of mean-field fully coupled forward-backward doubly stochastic differential equation arising naturally from the control problem, which is of interest in its own right. Some examples are provided to illustrate the applications of our results to control problems in the types of scalar interaction and first order interaction.

In this paper we study by probabilistic techniques the convergence of the value function for a two-scale, infinite-dimensional, stochastic controlled system as the ratio between the two evolution speeds diverges. The value function is represented as the solution of a backward stochastic differential equation (BSDE) that it is shown to converge towards a reduced BSDE. The noise is assumed to be additive both in the slow and the fast equations for the state. Some non degeneracy condition on the slow equation is required. The limit BSDE involves the solution of an ergodic BSDE and is itself interpreted as the value function of an auxiliary stochastic control problem on a reduced state space.

The problem of robust stabilization of a linear system leads to the classical ℋ︁∞ control problem. The same analysis applied to a nonlinear system leads to the problem of ensuring via output feedback that a nonlinear operator be Lipshitz continuous, with a prescribed Lipshitz modulus. We show that, in the same way as the ℋ︁∞ control problem is equivalent to a minimax control problem, the Lipshitz modulus control problem can be approached via a minimax team decision problem. This motivates us to re-visit a class of the so-called ‘static’ team decision problems for nonlinear dynamical control systems. Because of the ‘static’ character, signaling plays no role in that case, which is important for the equivalence with the Lipshitz modulus control problem. We show that under some conditions, a certainty equivalence principle applies that yields a practical solution to the team problem at hand. To reach that conclusion we must first investigate a ‘partial team’ problem where one of the team members has all the information. Copyright © 1999 John Wiley & Sons, Ltd.

A singular stochastic control problem with state constraints in two-dimensions is studied. We show that the value function is $C^1$ and its directional derivatives are the value functions of certain optimal stopping problems. Guided by the optimal stopping problem, we then introduce the associated no-action region and the free boundary and show that, under appropriate conditions, an optimally controlled process is a Brownian motion in the no-action region with reflection at the free boundary. This proves a conjecture of Martins, Shreve and Soner [SIAM J. Control Optim. 34 (1996) 2133--2171] on the form of an optimal control for this class of singular control problems. An important issue in our analysis is that the running cost is Lipschitz but not $C^1$. This lack of smoothness is one of the key obstacles in establishing regularity of the free boundary and of the value function. We show that the free boundary is Lipschitz and that the value function is $C^2$ in the interior of the no-action region. We then use a verification argument applied to a suitable $C^2$ approximation of the value function to establish optimality of the conjectured control.

We present the development and analysis of a reinforcement learning algorithm designed to solve continuous-space mean field game (MFG) and mean field control (MFC) problems in a unified manner. The proposed approach pairs the actor-critic (AC) paradigm with a representation of the mean field distribution via a parameterized score function, which can be efficiently updated in an online fashion, and uses Langevin dynamics to obtain samples from the resulting distribution. The AC agent and the score function are updated iteratively to converge, either to the MFG equilibrium or the MFC optimum for a given mean field problem, depending on the choice of learning rates. A straightforward modification of the algorithm allows us to solve mixed mean field control games. The performance of our algorithm is evaluated using linear-quadratic benchmarks in the asymptotic infinite horizon framework.

This chapter is an application of previous one. In Sect. 5.5, we have introduced the (controlled) Zakai equation, which is a stochastic linear parabolic equation with a Brownian adapted control process. By using the results in Chap. 5, we will study control problems for Zakai equations related to partially observable diffusions. The control problem for partially observable diffusions turns out to be a completely observable control problem on a Hilbert space, by using the unnormalized conditional probability density given by the Zakai equation (cf. Bensoussan A, Stochastic control of partially observable systems. Cambridge University Press, Cambridge/New York, 1992; Lions, J Commun PDE 8:1101–1134, 1983, I, II; Gozzi and Świech, J Funct Analy 172:466–510, 2000). Section 6.1 is devoted to the analysis of controlled Zakai equations. In Sect. 6.2, we formulate control problems for a system governed by Zakai equations, in the same way as in Chap. 2. When a control process \(\gamma (\cdot )\) is chosen, the cost on a time internal [T 0, t] is given by \(\int _{T_{0}}^{t}r(u^{\gamma (\cdot )}(s),\gamma (s))\,ds + F(u^{\gamma (\cdot )}(t))\), where \(u^{\gamma (\cdot )}(\cdot )\) is the response of \(\gamma (\cdot )\). By taking a suitable control process, we want to minimize (or maximize) the expectation of the cost. In Sect. 6.3 we formulate the DPP via the semigroup constructed from the value function, whose generator is related to the HJB equation on a Hilbert space. The viscosity solution of HJB equation is introduced following Gozzi and Świech (J Funct Analy 172:466–510, 2000) in Sect. 6.4. Example 6.1 makes explicitly the connection between controlled Zakai equations and control of partially observable diffusions.

We introduce a suitable backward stochastic differential equation (BSDE) to represent the value of an optimal control problem with partial observation for a controlled stochastic equation driven by Brownian motion. Our model is general enough to include cases with latent factors in mathematical finance. By a standard reformulation based on the reference probability method, it also includes the classical model where the observation process is affected by a Brownian motion (even in presence of a correlated noise), a case where a BSDE representation of the value was not available so far. This approach based on BSDEs allows for greater generality beyond the Markovian case, in particular our model may include path-dependence in the coefficients (both with respect to the state and the control), and does not require any nondegeneracy condition on the controlled equation. We use a randomization method, previously adopted only for cases of full observation, and consisting in a first step, in replacing the control by an exogenous process independent of the driving noise and in formulating an auxiliary (“randomized”) control problem where optimization is performed over changes of equivalent probability measures affecting the characteristics of the exogenous process. Our first main result is to prove the equivalence between the original partially observed control problem and the randomized problem. In a second step, we prove that the latter can be associated by duality to a BSDE, which then characterizes the value of the original problem as well.

We investigate a mean field optimal control problem obtained in the limit of the optimal control of large particle systems with forcing and terminal data which are not assumed to be convex. We prove that the value function, which is known to be Lipschitz continuous but not of class \(C^1\), in general, without convexity, is actually smooth in an open and dense subset of the space of times and probability measures. As a consequence, we prove a new quantitative propagation of chaos-type result for the optimal solutions of the particle system starting from this open and dense set.

We prove a convergence theorem for a family of value functions associated with stochastic control problems whose cost functions are defined by backward stochastic differential equations. The limit function is characterized as a viscosity solution to a fully nonlinear partial differential equation of second order. The key assumption we use in our approach is shown to be a necessary and sufficient assumption for the homogenizability of the control problem. The results generalize partially homogenization problems for Hamilton–Jacobi–Bellman equations treated recently by Alvarez and Bardi by viscosity solution methods. In contrast to their approach, we use mainly probabilistic arguments, and discuss a stochastic control interpretation for the limit equation.

Numerical analysis of a free-boundary singular control problem in financial economics

We consider a dynamic scheduling problem for parallel server systems. J. M. Harrison has proposed a scheme for using diffusion control problems to approximately solve such control problems for heavily loaded systems. This approach has been very successfully used in the special case when the diffusion control problem can be reduced to an equivalent one for a one-dimensional workload process. However, it remains a challenging open problem to make substantial progress on using Harrison’s scheme when the workload process is more than one-dimensional. Here we present some new structural results concerning the diffusion control problem for parallel server systems with arbitrary workload dimension. Specifically, we prove that a certain server-buffer graph associated with a parallel server system is a forest of trees. We then exploit this graphical structure to prove that there exists a matrix, used to define the workload process, that has a block diagonal-like structure. An important feature of this matrix is that, except when the workload is one-dimensional, this matrix is frequently different from a choice of workload matrix proposed by Harrison. We demonstrate that our workload matrix simplifies the structure of the control problem for the workload process by proving that when the original diffusion control problem has linear holding costs, the equivalent workload formulation also has a linear cost function. We also use this simplification to give sufficient conditions for a certain least control process to be an optimal control for the diffusion control problem with linear holding costs. Under these conditions, we propose a continuous review threshold-type control policy for the original parallel server system that exploits pooling of servers within trees in the server-buffer graph and uses non-basic activities connecting different trees in a critical manner. We call this partial pooling. We conjecture that this threshold policy is asymptotically optimal in the heavy traffic limit. We illustrate the solution of the diffusion control problem and our proposed threshold control policy for a three-buffer, three-server example.

The classical Method of Successive Approximations (MSA) is an iterative method for solving stochastic control problems and is derived from Pontryagin’s optimality principle. It is known that the MSA may fail to converge. Using careful estimates for the backward stochastic differential equation (BSDE) this paper suggests a modification to the MSA algorithm. This modified MSA is shown to converge for general stochastic control problems with control in both the drift and diffusion coefficients. Under some additional assumptions the rate of convergence is shown. The results are valid without restrictions on the time horizon of the control problem, in contrast to iterative methods based on the theory of forward-backward stochastic differential equations.