Optimal Encoders Maximizing Directed Information of Channels with Memory and Feedback: Stochastic Control and Dynamic Programming

This paper studies a class of renewable resource allocation problems for the processing of dynamically arriving tasks with deterministic deadlines. This class of problems has many applications, however, it conforms to neither the standard resource allocation model nor the standard optimal control model. A new problem formulation has to be developed and analyzed to provide a satisfactory solution. In this paper, a new formulation is presented. By augmenting the state variabls, the problem is converted into a Markovian decision problem. It can then be treated, at least in principle, by using the stochastic dynamic programming (SDP) method. However, since the system dynamics involves the evolution of sets (of tasks and resources), the implementation of the dynamic programming equation is by no means straightforward. For a problem with infinite planning horizon, the optimal strategy is shown to be stationary under mild conditions. An SDP algorithm based on a successive approximation technique is developed to obtain the optimal stationary strategy. The implementation of the algorithm employs a special coding scheme to handle set variables, and utilizes a dominance property for computational efficiency. Effects of key system parameters on optimal decisions are investigated and analyzed through numerical examples. As the computational complexity of the algorithm is of exponential increase, practical applications of the algorithm is limited to problems of moderate size. Two heuristic rules are therefore investigated and compared to the optimal policy.

In this paper, the application of the dynamic programming approach to constrained stochastic control problems with expected value constraints is demonstrated. Specifically, two such problems are analyzed using this approach. The problems analyzed are the problem of minimizing a discounted cost infinite horizon expectation objective subject to an identically structured constraint, and the problem of minimizing a discounted cost infinite horizon minimax objective subject to a discounted expectation constraint. Using the dynamic programming approach, optimality equations, which are the chief contribution of this paper, are obtained for these problems. In particular, the dynamic programming operators for problems with expectation constraints differ significantly from those of standard dynamic programming and problems with worst-case constraints. For the discounted cost infinite horizon cases, existence and uniqueness of solutions to the dynamic programming equations are explicitly shown by using the Banach fixed point theorem to show that the corresponding dynamic programming operators are contractions. The theory developed is illustrated by numerically solving the constrained stochastic control dynamic programming equations derived for simple example problems. The example problems are based on a two-state Markov model that represents an error prone system that is to be maintained.

In this article, we consider secure transmission of a deterministic vector parameter from a transmitter to an intended receiver in the presence of a smart eavesdropper. The aim is to determine the optimal power allocation and optimal linear encoding strategies at the transmitter to maximize the estimation performance at the intended receiver under constraints on the estimation performance at the eavesdropper and on the transmit power. First, the A-optimality criterion is adopted by utilizing the Cramér-Rao lower bound as the estimation performance metric, and the optimal power allocation and optimal linear encoding strategies are characterized theoretically. Then, corresponding to the D-optimality criterion, the determinant of the Fisher information matrix is considered as the estimation performance metric. It is shown that the optimal linear encoding and optimal power allocation strategies lead to the same solution for this criterion. In addition, extensions of the theoretical results are provided to cases with statistical knowledge of systems parameters. Numerical examples are provided to investigate the optimal power allocation and optimal linear encoding strategies in different scenarios.

A computational approach is taken to solve the optimal nonlinear stochastic control problem. The approach is to systematically solve the stochastic dynamic programming equations forward in time, using a nested stochastic approximation technique. Although computationally intensive, this provides a straightforward numerical solution for this class of problems and provides an alternative to the usual dimensionality problem associated with solving the dynamic programming equations backward in time. It is shown that the cost degrades monotonically as the complexity of the algorithm is reduced. This provides a strategy for suboptimal control with clear performance/computation tradeoffs. A numerical study focusing on a generic optimal stochastic adaptive control example is included to demonstrate the feasibility of the method.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

We consider the stochastic optimal control problem of McKean−Vlasov stochastic differential equation where the coefficients may depend upon the joint law of the state and control. By using feedback controls, we reformulate the problem into a deterministic control problem with only the marginal distribution of the process as controlled state variable, and prove that dynamic programming principle holds in its general form. Then, by relying on the notion of differentiability with respect to probability measures recently introduced by [P.L. Lions, Cours au Collège de France: Théorie des jeux à champ moyens, audio conference 2006−2012], and a special Itô formula for flows of probability measures, we derive the (dynamic programming) Bellman equation for mean-field stochastic control problem, and prove a verification theorem in our McKean−Vlasov framework. We give explicit solutions to the Bellman equation for the linear quadratic mean-field control problem, with applications to the mean-variance portfolio selection and a systemic risk model. We also consider a notion of lifted viscosity solutions for the Bellman equation, and show the viscosity property and uniqueness of the value function to the McKean−Vlasov control problem. Finally, we consider the case of McKean−Vlasov control problem with open-loop controls and discuss the associated dynamic programming equation that we compare with the case of closed-loop controls.

Nonlinear stochastic optimal control of Preisach hysteretic systems

This paper introduces a new type of dynamic programming equations for optimal control problems with performance criteria involving multiple integrals. The main novel feature of the multi-time dynamic programming equations, relative to the standard Hamilton-Jacobi equation, is that they form a Goursat-Darboux problem (i.e. boundary data are needed on two lines, or on planes or hyperplanes), whereas Hamilton-Jacobi equations form a Cauchy problem (initial value problem). In the case of performance criteria involving multiple integrals with quadratic integrands, the new equations lead to multi-time variants of the Riccati equation.

In this chapter, we assume that the filtration \(\mathbb{F}\) is the \(\mathbb{P}\)−augmentation of the canonical filtration of the Brownian motion W. This restriction is only needed in order to simplify the presentation of the proof of the dynamic programming principle.

The stochastic optimal nonlinear control of coupled adjacent building structures is studied based on the stochastic dynamical programming principle and the stochastic averaging method. The coupled structures with control devices under random seismic excitation are first condensed to form a reduced-order structural model for the control analysis. The stochastic averaging method is applied to the reduced model to yield stochastic differential equations for structural modal energies as controlled diffusion processes. Then a dynamical programming equation for the energy processes is established based on the stochastic dynamical programming principle, and solved to determine the optimal nonlinear control law. The seismic response mitigation of the coupled structures is achieved through the structural energy control and the dimension of the optimal control problem is reduced. The seismic excitation spectrum is taken into account according to the stochastic dynamical programming principle. Finally, the nonlinear controlled structural response is predicted by using the stochastic averaging method and compared with the uncontrolled structural response to evaluate the control efficacy. Numerical results are given to demonstrate the response mitigation capabilities of the proposed stochastic optimal control method for coupled adjacent building structures.

Series Solution of Stochastic Dynamic Programming Equations

1 - THE STOCHASTIC VIEW OF ECONOMICS

This paper analyzes the fundamental limit of the strategic semantic communication problem in which a transmitter obtains a limited number of indirect observations of an intrinsic semantic information source and can then influence the receiver’s decoding by sending a limited number of messages over an imperfect channel. The transmitter and the receiver can have different distortion measures and can make rational decisions about their encoding and decoding strategies, respectively. The decoder can also have some side information (e.g., background knowledge and/or information obtained from previous communications) about the semantic source to assist its interpretation of the semantic information. We focus particularly on the case that the transmitter can commit to an encoding strategy and study the impact of the strategic decision making on the rate distortion of semantic communication. Three equilibrium solution concepts including the optimal Stackelberg equilibrium, robust Stackelberg equilibrium, as well as Nash equilibrium are studied and compared. The optimal encoding and decoding strategy profiles under various equilibrium solutions are derived. We prove that committing to an encoding strategy cannot always bring benefit to the encoder. We provide a feasible condition under which committing to an encoding strategy can always reduce the distortion of semantic communication. We consider an example with a dictionary-based semantic information source to verify our observation.

Reservoir management is a highly complex problem, which includes the uncertainty in the inflows as well as in the objectives. Stochastic dynamic programming (SDP) has been widely used to develop the optimal operating policies of multipurpose reservoirs. In the recent years, fuzzy optimization has been used to incorporate the uncertainty caused due to the imprecise nature of objectives. In the present study, a comparison is made between a conventional SDP model and fuzzy dynamic programming models for optimal operation of a multipurpose reservoir. The two models are developed for Hirakud Reservoir on River Mahanadi in the State of Orissa in India. The performance of the reservoir is evaluated under the operating policies derived from these models through simulation. The results indicate that the fuzzy dynamic programming model performs better in achieving the flood control objective and thus in the overall performan1ce of reservoir.

Compared with the conventional energy power generation investments, the uncertainty of wind power investment will bring greater risk. For wind power investors, wind power investment strategy-makers, even for wind power equipment manufacturers, all urgently need decision-support tools to minimize the risk and obtain optimal investment opportunities. In this paper, firstly we analyze the current problems and risks of wind power industry in China. Secondly, based on stochastic theory and dynamic programming methods, the paper establishes model about wind power investment decision-making. It considers the wind power project investment uncertainties risk factors. Example analysis shows that the model can be used for wind power developers to choose the right investment opportunities and to provide decision-making reference.

The increased water demand and the increased capital construction costs of water resource development projects force engineers and water planners to utilize efficiently the existing and proposed water resources systems. Hence, it is considered necessary to find the optimum real time operating rules for systems of multiple reservoirs.Stochastic dynamic programming (SDP) has been used as the tool for deriving the real time operating rules for a system of multiple reservoirs, in preference to simulation analysis and linear programming. This is because the operation of the storages is considered as a sequential decision problem and dynamic programming is well suited to handle sequential, multi-period type decision problems. Further, the stochastic nature of streamflow is considered explicitly through a multivariate distribution for streamflow. The method theoretically produces the global optimum solution.The computational difficulties in terms of computer time and memory space have prohibited the realistic application of this method to systems of multiple reservoirs in the past. In this study, two methods have been introduced to reduce the computational difficulties of the method, thus making SDP more attractive to use in the derivation of the real time operating rules for a system of multiple reservoirs.The first method, the assumption of cross correlation of streamflow, reduces the number of states in the SDP method which reduces computer time and memory space. Further, it reduces the number of conditional probability estimates of streamflow which are used in recursive relation of the SDP algorithm. The second method, the “corridor” approach, reduces computer time involved in the SDP method by eliminating the computation of infeasible and/or inferior (suboptimal) solutions. These two methods have resulted in significant savings in the computer effort of the SDP method.The conditional probabilities of streamflow are computed by considering the hypervolume under the conditional probability density function between the integration limits using the computationally efficient Gaussian Legendre quadrature method. The conditional probability density function is derived from the historical streamflow data.Real time operating rules are expressed as a function of the storage volume(s) at the beginning of the current time interval and the forecast inflow into the storage(s) during the current time interval.The Melbourne water supply system is used as the example. The derived operating rules have been tested by operating a simulation model and the conclusion is made that the SDP method is a satisfactory technique for objectively deriving real time operating rules for a system of multiple reservoirs.Finally, a combined stochastic dynamic programming – statistical disaggregation approach is introduced to determine the real time operating rules for a system of multiple reservoirs where operational data are available. Releases from individual storages are a function of the total system storage volume and inflow. The method uses SDP to determine the release rules from the equivalent single reservoir of the multiple reservoir system; these releases are then disaggregated to individual releases using the statistical release disaggregation model whose parameters are derived from the historical operational data. The approach was used to derive real time operating rules for the Melbourne water supply system and the operating rules were validated using a simulation model of the system.