Optimal control of the two-state Markov process in discrete time

If we follow an asexually reproducing population through time, then the amount of time that has passed since the most recent common ancestor (MRCA) of all current individuals lived will change as time progresses. The resulting “MRCA age” process has been studied previously when the population has a constant large size and evolves via the diffusion limit of standard Wright–Fisher dynamics. For any population model, the sample paths of the MRCA age process are made up of periods of linear upward drift with slope +1 punctuated by downward jumps. We build other Markov processes that have such paths from Poisson point processes on ℝ++×ℝ++ with intensity measures of the form λ⊗μ where λ is Lebesgue measure, and μ (the “family lifetime measure”) is an arbitrary, absolutely continuous measure satisfying μ((0, ∞))=∞ and μ((x, ∞))<∞ for all x>0. Special cases of this construction describe the time evolution of the MRCA age in (1+β)-stable continuous state branching processes conditioned on nonextinction—a particular case of which, β=1, is Feller’s continuous state branching process conditioned on nonextinction. As well as the continuous time process, we also consider the discrete time Markov chain that records the value of the continuous process just before and after its successive jumps. We find transition probabilities for both the continuous and discrete time processes, determine when these processes are transient and recurrent and compute stationary distributions when they exist. Moreover, we introduce a new family of Markov processes that stands in a relation with respect to the general (1+β)-stable continuous state branching process and its conditioned version that is similar to the one between the family of Bessel-squared diffusions and the unconditioned and conditioned Feller continuous state branching process.

We considered the issues associated with methods for solving the problems of representation of continuous Markov processes in discrete-time by the example of radar signals, which are subject to the simultaneous effect of additive and multiplicative noise described by stochastic differential equations. We considered the algorithm, which allows us to carry out computer simulation of fading amplitude, phase and information parameter of the signal in discrete observation time using the example of a continuous frequency-modulated signal, the amplitude of which adheres to the Nakagami distribution that is typical for radar signals under simultaneous effect of additive and multiplicative noise. It is shown that with the help of computer simulation it is possible to create not only optimal algorithms, but also their corresponding optimal structural diagrams, which makes it possible to elevate the processing of radar signals to higher standards.

A probabilistic model for the slow shearing motion of spherical particles dominated by dry inter-particle frictional forces is proposed. The particles can move by sliding, rolling, solid body rotation and jumping following the criterion of minimum energy dissipation. The continuous deformation of particle assembly is regarded as a Markov process in discrete time and continuous space. The components of the contact angle between two neighboring particles are taken as the primary random variables. The coefficients of the probabilistic governing equation are determined from the rate of energy dissipation and micromechanical analysis of particle motion. The inter-particle sliding and rolling friction laws are input data. The constitutive relationships, apparent viscosity and internal friction angle of the particulate material as a whole are obtained by integration. The validity of the proposed model is verified by comparing the model predictions to analytical solutions and available experimental results.

A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a given convex family of distributions depending on the present state. The immediate cost is prescribed for each choice and it is required to minimise the average expected cost over an infinite future. The paper considers a special case of this general problem and provides the foundation for a general solution. The main result is that an optimal policy exists if each state of the system can be reached with positive probability from any other state by choosing a suitable policy.

Probabilistic micromechanical model for two-dimensional slow particulate flow: Dry friction

The rapid development of UAV technologies and the wide access to them by various consumers give rise to a new class of threats. To neutralize them, complex UAV protection systems are created, which include passive and active measuring systems. As a passive system, a wireless sensor network is used, which allows to determine location of UAV with high accuracy based on TDOA measurements. However, in real conditions, abnormal (rough) measurement results may appear that lead to divergence of Kalman filtering algorithms for UAV movement parameters.On the basis of a mathematical apparatus of the mixed Markov processes in discrete time, a quasi-optimal algorithm of adaptive estimation of UAV movement parameters based on the TDOA-measurement of the sensor network in the presence of abnormal measurement errors was synthesized. The efficiency analysis of obtained algorithm and its comparison with algorithm of Kalman filtering are performed using the statistical simulation..

Extended piecewise Markov processes in discrete time

To increase the detection efficiency of moving targets masked by interfering reflections, radars are widely used, in which target spatial coordinates and also range rate are measured. When solving the radar data processing problems, the presence of target range rate measurements increases the target tracking efficiency in the presence of false plots, and can significantly improve the tracking quality, especially with intensive target maneuvering. In the work on the mathematical apparatus basis of the mixed Markov processes in discrete time, a quasi-optimal adaptive algorithm for tracking maneuvering targets in a difficult jamming environment for a multifunctional radar with range rate measurement is developed. Using statistical computer modeling, a comparative analysis of tracking algorithms with and without measurement of range rate in the same interference environment at different data rates is performed.

In this paper, we study the classification problem of discrete time and continuous time Markov processes with a tree structure. We first show some useful properties associated with the fixed points of a nondecreasing mapping. Mainly we find the conditions for a fixed point to be the minimal fixed point by using fixed point theory and degree theory. We then use these results to identify conditions for Markov chains of M/G/1 type or GI/M/1 type with a tree structure to be positive recurrent, null recurrent, or transient. The results are generalized to Markov chains of matrix M/G/1 type with a tree structure. For all these cases, a relationship between a certain fixed point, the matrix of partial differentiation (Jacobian) associated with the fixed point, and the classification of the Markov chain with a tree structure is established. More specifically, we show that the Perron-Frobenius eigenvalue of the matrix of partial differentiation associated with a certain fixed point provides information for a complete classification of the Markov chains of interest.

A derivation of a law of large numbers for the highest-scoring matching subsequence is given. Let Xk, Yk be i.i.d. q=(q(i)) i∊S letters from a finite alphabet S and v=(v(i)) i∊S be a sequence of non-negative real numbers assigned to the letters of S. Using a scoring system similar to that of the game Scrabble, the score of a word w=i 1 · ·· im is defined to be V(w)=v(i 1) + · ·· + v(im ). Let Vn denote the value of the highest-scoring matching contiguous subsequence between X 1 X 2 · ·· Xn and Y 1 Y 2 · ·· Yn. In this paper, we show that Vn/K log(n) → 1 a.s. where K ≡ K(q , v). The method employed here involves ‘stuttering’ the letters to construct a Markov chain and applying previous results for the length of the longest matching subsequence. An explicit form for β ∊Pr(S), where β (i) denotes the proportion of letter i found in the highest-scoring word, is given. A similar treatment for Markov chains is also included. Implicit in these results is a large-deviation result for the additive functional, H ≡ Σ n &amp;lt; τ v(Xn ), for a Markov chain stopped at the hitting time τ of some state. We give this large deviation result explicitly, for Markov chains in discrete time and in continuous time.

We develop a general framework for stationary marked point processes in discrete time. We start with a careful analysis of the sample paths. Our initial representation is a sequence $$\{(t_j,k_j): j\in {\mathbb {Z}}\}$$ of times $$t_j\in {\mathbb {Z}}$$ and marks $$k_j\in {\mathbb {K}}$$ , with batch arrivals (i.e., $$t_j=t_{j+1}$$ ) allowed. We also define alternative interarrival time and sequence representations and show that the three different representations are topologically equivalent. Then, we develop discrete analogs of the familiar stationary stochastic constructs in continuous time: time-stationary and point-stationary random marked point processes, Palm distributions, inversion formulas and Campbell’s theorem with an application to the derivation of a periodic-stationary Little’s law. Along the way, we provide examples to illustrate interesting features of the discrete-time theory.

dtsdPBC extends the well-known algebra of parallel processes, Petri box calculus (PBC), by incorporating discrete time stochastic and deterministic delays. To analyze performance in this extended calculus, the underlying semi-Markov chains, and the related (complete) and reduced discrete time Markov chains of the process expressions are built. The semi-Markov chains are extracted using the embedding method, which constructs the embedded discrete time Markov chains and calculates the sojourn time distributions in the states. The reductions of the discrete time Markov chains are obtained through the elimination method, which removes the vanishing states (those with zero sojourn times) and recalculates the transition probabilities among the tangible states (those with positive sojourn times). We prove that the reduced semi-Markov chain coincides with the reduced discrete time Markov chain, by demonstrating that an additional embedding into the reduced semi-Markov chain is needed for the reduced embedded discrete time Markov chain to match the embedded reduced discrete time Markov chain, and by comparing the respective sojourn times.

Optimal Hedging in Discrete and Continuous Time

ABSTRACTIn epidemic modelling, the emergence of a disease is characterized by the low numbers of infectious individuals. Environmental randomness impacts outcomes such as the outbreak or extinction of the disease in this case. This randomness can be accounted for by modelling the system as a continuous time Markov chain, . The probability of extinction given some initial state is the probability of hitting a subset of the state space associated with extinction for the initial state. This hitting probability can be studied by passing to the discrete time Markov chain (DTMC), . An approach is presented to approximate a DTMC on a countably infinite state space by a DTMC on a finite state space for the purpose of solving general hitting problems. This approach is applied to approximate the probability of disease extinction in an epidemic model. It is also applied to evaluate a heterogeneous disease control strategy in a metapopulation.

The prediction of numbers of newborns is an important issue in hospital management. Relying on the inherent non-aftereffect property, discrete-time Markov chain (DTMC) is a candidate for solving the problem. But the classical DTMC is unable to handle the uncertainty of states, especially when the state space is not discrete, which would lead to instable predicted results. In order to overcome the limitation of the existing DTMC model, a belief Markov chain (BMC) model is proposed by synthesizing the classical DTMC and Dempster-Shafer theory effectively. Depending on the advantages of Dempster-Shafer theory in expressing uncertainty, the proposed BMC model is capable of dealing with various uncertainties, which improves and perfects the classical DTMC model. An illustrative example demonstrates the effectiveness of the proposed model. Moreover, a comparison between the proposed BMC model and the classical and fuzzy states modified DTMC models is given to show the superiority of the proposed model against the other two. Finally, the stability of the proposed model has been proven.