<title>A stochastic grid filter for multi-target tracking</title>

Numerical solutions of stochastic control problems: Markov chain approximation methods

Applications of Markov chain approximation methods to optimal control problems in economics

Continuous approximation of collective system behaviour: A tutorial

In this paper, a numerical algorithm is presented to solve stochastic optimal control problems via the Markov chain approximation method. This process is based on state and time spaces discretization followed by a backward iteration technique. First, the original controlled process by an appropriate controlled Markov chain is approximated. Then, the cost functional is appropriate for the approximated Markov chain. Also, the finite difference approximations are used to the construction of locally consistent approximated Markov chain. Furthermore, the coefficients of the resulting discrete equation can be considered as the desired transition probabilities and interpolation interval. Finally, the performance of the presented algorithm on a test case with a well-known explicit solution, namely the Merton’s portfolio selection model, is demonstrated.

We deal with a very useful numerical method for both controlled and uncontrolled queuing and multiplexing type systems. The basic idea starts with a heavy traffic approximation, but it is shown that the results are very good even when working far from the heavy traffic regime. The underlying numerical method is a version of what is known as the Markov chain approximation method. It is a powerful methodology for controlled and uncontrolled stochastic systems, which can be approximated by diffusion or reflected diffusion type systems, and has been used with success on many other problems in stochastic control. We give a complete development of the relevant details, with an emphasis on multiplexing and particular queueing systems. The approximating process is a controlled or uncontrolled Markov chain which retains certain essential features of the original problem. This problem is generally substantially simpler than the original physical problem, and there are associated convergence theorems. The non-classical associated ergodic cost problem is derived, and put into a form such that reliable and good numerical algorithms, based on multigrid type ideas, can be used. Data for both controlled and uncontrolled problems shows the value of the method.

Motivated by applications arising in networked systems, this work examines controlled regime-switching systems that stem from a mean-variance formulation. A main point is that the switching process is a hidden Markov chain. An additional piece of information, namely, a noisy observation of switching process corrupted by white noise is available. We focus on minimizing the variance subject to a fixed terminal expectation. Using the Wonham filter, we convert the partially observed system to a completely observable one first. Since closed-form solutions are virtually impossible be obtained, a Markov chain approximation method is used to devise a computational scheme. Convergence of the algorithm is obtained. A numerical example is provided to demonstrate the results.

Convergence Rate Analysis for the Continuous-Time Markov Chain Approximation of Occupation Time Derivatives and Asian Option Greeks

Approximate Model Checking of Stochastic Hybrid Systems

Dealing with a stochastic dynamical system with state vector X(t) satisfying a nonlinear Langevin equation of the type $$dx = f(x)dt + \sigma (x)dW(t),$$ we may only be interested in qualitative changes and their statistics. Typical qualitative changes are the switch from one domain of attraction to a neighbouring one or in the case of population dynamics the extinction or introduction of a species. For that purpose we introduce the Markov chain approximation. For a precise mathematical description of the connection between such a Markov chain and the stochastic dynamical system we refer to Chap. 6 of Freidlin and Wentzell (1984). Here we work out two examples. One modelling the preferent states of the atmospheric circulation and one dealing with the occupation rate of a patch of land by some biological species.

We extend the numerical methods of [Kushner, H.J. and Dupuis, P., 1992, Numerical Methods for Stochastic Control Problems in Continuous Time, 2nd ed., 2001 (Berlin and New York: Springer Verlag], known as the Markov chain approximation methods, to controlled general nonlinear delayed reflected diffusion models. Both the path and the control can be delayed. For the no-delay case, the method covers virtually all models of current interest. The method is robust, the approximations have physical interpretations as control problems closely related to the original one, and there are many effective methods for getting the approximations, and for solving the Bellman equation for low-dimensional problems. These advantages carry over to the delay problem. It is shown how to adapt the methods for getting the approximations, and the convergence proofs are outlined for the discounted cost function. Extensions to all of the cost functions of current interest as well as to models with Poisson jump terms are possible. The paper is particularly concerned with representations of the state and algorithms that minimize the memory requirements.

In this paper, we consider a class of continuous-time, continuous-space stochastic optimal control problems. Using the Markov chain approximation method and recent advances in sampling-based algorithms for deterministic path planning, we propose a novel algorithm called the incremental Markov Decision Process to incrementally compute control policies that approximate arbitrarily well an optimal policy in terms of the expected cost. The main idea behind the algorithm is to generate a sequence of finite discretizations of the original problem through random sampling of the state space. At each iteration, the discretized problem is a Markov Decision Process that serves as an incrementally refined model of the original problem. We show that with probability one, (i) the sequence of the optimal value functions for each of the discretized problems converges uniformly to the optimal value function of the original stochastic optimal control problem, and (ii) the original optimal value function can be computed efficiently in an incremental manner using asynchronous value iterations. Thus, the proposed algorithm provides an anytime approach to the computation of optimal control policies of the continuous problem. The effectiveness of the proposed approach is demonstrated on motion planning and control problems in cluttered environments in the presence of process noise.

Markov chain approximations to singular stable-like processes

This paper discusses the problem of state estimation for non‐linear stochastic uncertain systems with time‐correlated additive noise and packet dropout compensations. Both process noise and measurement noise are additive forms of noise that can be described as Gaussian first‐order Markov processes. The stochastic uncertainty of the system is described by synchronous correlated white multiplicative noise. When the measurement information from the current epoch is lost, a compensation strategy based on measurement information prediction is used. First, the state information, process noise, and measurement noise are augmented to enable the construction of a new vector to be estimated, and the filter, predictor, and smoother of the system are designed using a Gaussian iterative estimation algorithm based on innovation analysis. The multiplicative noise in the process equation is estimated in real time. Second, a numerical implementation of the proposed algorithm is derived based on the third‐degree spherical‐radial Cubature Kalman filter(CKF). Finally, the proposed algorithm is compared with an extended Kalman filter (EKF) based on Taylor series expansion. The effectiveness of the proposed algorithm is verified by comparing the simulation results.

Approximate Abstractions of Stochastic Hybrid Systems

Efficient simulation of generalized SABR and stochastic local volatility models based on Markov chain approximations