Dynamic feedback control of stochastic-parameter systems

In this work, an approach to stabilizing compensator design is presented that is suitable for a large class of discrete-time non-linear stochastic systems with incomplete state information. The non-linearity is contained either in the state or the measurement equation and it involves a white-noise vector sequence. The proposed compensator is a combination of a constant gain observer and a constant feedback control gain. The stabilizability conditions depend upon the system and chosen design parameter characteristics and are given for both mean-square and almost sure stochastic boundedness and stability of the controlled system. A specific subclass of these systems, subject to certain control laws, is shown to possess stabilizability conditions that can be tested a priori.

This paper investigates an echo state network (ESN) (Jaeger, 2001 and Maass and Markram, 2002) architecture as the approximation of the Q-function for temporally dependent rewards embedded in a linear dynamical system, the mass-spring-damper (MSD). This problem has been solved utilizing feed-forward neural networks (FNN) when all state information necessary to specify the dynamics is provided as input (Kretchmar, 2000). Time-delayed neural networks (TDNN) solve this problem with finite-size windows of incomplete state information. Our research demonstrates that the ESN architecture represents the Q-function of the MSD system given incomplete state information as well as current feed forward neural networks given either perfect state or a temporally-windowed, incomplete state vector. The remainder of this paper is organized as follows. We introduce basic concepts of reinforcement learning and the echo state network architecture. The MSD system simulation is defined in section IV. Experimental results for learning state quality given incomplete state information are presented in section V. Results for learning estimates of all future state qualities for incomplete state information is presented in section VI. Section VII discusses the potential of the ESN for use in reinforcement learning and provides current and future directions of research.

Closed-Loop Strategy for Stackelberg Game Problem with Incomplete Information Structures

Rotating stall and surge have been stabilized in a transonic single-stage axial compressor using active feedback control. The control strategy is to sense upstream wall static pressure patterns and feed back the signal to an annular array of twelve separately modulated air injectors. At tip relative Mach numbers of 1.0 and 1.5 the control achieved 11 and 3.5 percent reductions in stalling mass flow, respectively, with injection adding 3.6 percent of the design compressor mass flow. The aerodynamic effects of the injection have also been examined. At a tip Mach number, Mtip, of 1.0, the stall inception dynamics and effective active control strategies are similar to results for low-speed axial compressors. The range extension was achieved by individually damping the first and second spatial harmonics of the prestall perturbations using constant gain feedback. At a Mtip of 1.5 (design rotor speed), the prestall dynamics are different than at the lower speed. Both one-dimensional (surge) and two-dimensional (rotating stall) perturbations needed to be stabilized to increase the compressor operating range. At design speed, the instability was initiated by approximately ten rotor revolutions of rotating stall followed by classic surge cycles. In accord with the results from a compressible stall inception analysis, the zeroth, first, and second spatial harmonics each include more than one lightly damped mode, which can grow into the large amplitude instability. Forced response testing identified several modes traveling up to 150 percent of rotor speed for the first three spatial harmonics; simple constant gain control cannot damp all of these modes and thus cannot stabilize the compressor at this speed. A dynamic, model-based robust controller was therefore used to stabilize the multiple modes that comprise the first three harmonic perturbations in this transonic region of operation.

An unconstrained stochastic optimization problem involving a discrete-time linear process with a normally distributed initial condition and subject to additive gaussian state and measurement noise is formulated in terms of a quite general finite horizon, discrete-time quadratic cost criterion and solved when there is either complete or incomplete state information. It is shown that both the stochastic sampled-data optimal tracker and the stochastic sampled-data optimal regulator are special cases of this problem. A breakdown of the minimum cost for both sampled-data controllers is given.

Designing mobile robots that navigate indoor environments autonomously is known to be a difficult problem. A critical issue is in the formulation of robust motion control algorithms capable of reliably sensing partial or incomplete information from the environment and using this information to choose appropriate actions to achieve its designed goals. As an example, suppose we wish to deploy a mobile robot that autonomously patrols defined locations at a hazardous high-security facility. The robot must maintain accurate knowledge of its location, while using sensory data to recognize objects and obstacles in its immediate vicinity. Its task is to inspect the desired locations within a defined time period and provide real-time data in the event of an incident. The problem is thus to choose appropriate actions that result in accomplishing the patrol in a minimal amount of time in the partially structured environment. To solve this problem we adopt the Partially Observable Markov Decision Processes (POMDP) formalism to find near-optimal and efficient policies that provides a description the robot's motion in environments with incomplete state information. POMDP is a generalization of Markov Decision Processes (MDPs). It models a system as a coupling of an agent/decision maker (robots in our case) and an environment. We also present a methodology called Goal-Specific Representation (GSR) to reduce the size of the state-space for computational efficiency and propose an extension to the methodology.

In this paper, we will derive complete parametrization of the H∞ FI controller and that of the H∞ sdtate feedback (SF) controller. And we will determine the obtained dynamical free parameters so that they minimizes an H2 control performance and will show the condition when the whole controller becomes a constant feedback gain other than the central solution F∞.

In heterogeneous networks, interference management is very important since the interference due to spectrum-sharing can significantly decrease the overall performance. In the existing work, various resource allocation methods are proposed to either improve energy efficiency or alleviate interference. Very little research has addressed their joint interaction. Furthermore, most existing work assumes that all of the channel state information is known, which might not be realistic. In this paper, we propose a game-theoretical scheme using energy-efficient resource allocation and interference pricing for an interference-limited environment in two-tier heterogeneous networks without complete channel state information. We formulate the problems of resource allocation and interference management as a two-stage Stackelberg game with incomplete information. A backward induction method is used to analyze the proposed game, since it can capture the sequential dependencies of the decisions in the stages of the game.

Numerous phenomena in the theory of automatic regulation, mechanics, radiophysics, biology, immunology, economics, robototechnics, etc. need for theiccorrect qualitative and quantitative description to use equations with time lag. One problem which here arises is connected with states estimate of stochastic hereditary systems [1]. This paper is a survey of some results and problems of filtering, interpolation and smoothing for arbitrary Gaussian stochastic processes for the case of linear observations with delays. In particular, the estimate problems of solutions of linear stochastic differential equations with delays and integral equations of Volterra are considered. Besides new method for solution of optimal control problem with incomplete state information is adduced. The survey is based on the papers [2-14]

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming “beliefs”—optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN’s learned representation encodes belief information, but only when the RNN’s capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

Optimal control of Markov processes with incomplete state information

Actor-critic reinforcement learning for the feedback control of a swinging chain

Aiming to monitor wear condition of milling cutters in time and provide tool change decisions to ensure manufacturing safety and product quality, a tool wear monitoring model based on Bagging-Gradient Boosting Decision Tree (Bagging-GBDT) is proposed. In order to avoid incomplete tool state information contained in a single domain feature parameter, a multi-domain combination method is used to extract candidate characteristic parameter sets from time domain, frequency domain, and time–frequency domain. Then top 21 significant features are screened by eXtreme Gradient Boosting selection method. Synthetic Minority Oversampling Technique technology is integrated during feature selection to overly sample feature vectors, so that wear condition categories can be well balanced. Bagging idea is then introduced for parallel calculation of the gradient boosting decision tree and to improve its generalization ability. A Bagging-GBDT milling cutter wear condition prediction model is constructed and verified by public ball-end milling data set. Experiments show that random features and training samples selection can effectively improve prediction performance and generalization ability of prediction model. Our Bagging-GBDT model gains F1 score of 0.99350, which is 0.2% and 13.2% higher than the random forest algorithm and basic GBDT model, respectively.

We study opportunistic multiuser scheduling in downlink networks with Markov-modeled outage channels. We consider the scenario that the scheduler does not have full knowledge of the channel state information, but instead estimates the channel state by exploiting the memory inherent in the Markov channels along with Automatic-Repeat-reQues-styled-styled feedback from the scheduled users. Opportunistic scheduling is optimized in two stages: 1) channel estimation and rate adaptation are performed to maximize the short-term throughput, i.e., the successful transmission rate of the scheduled user in the current slot and 2) user scheduling is performed, based on the short-term throughput, to maximize the overall long-term sum-throughput of the downlink. The scheduling problem is a partially observable Markov decision process with the classic exploitation versus exploration tradeoff that is difficult to quantify. We, therefore, study the problem in the framework of restless multiarmed bandit processes, and perform a Whittle’s indexability analysis. Whittle’s indexability is traditionally known to be hard to establish and the index policy derived based on Whittle’s indexability is known to have optimality properties in various settings. We show that the problem of downlink scheduling under imperfect channel state information is Whittle indexable and derive the Whittle’s index policy in closed form. Through extensive numerical experiments, we show that the Whittle’s index policy has near-optimal performance and is robust against various imperfections in channel state feedback. Our work reveals that, under incomplete channel state information, exploiting channel memory for opportunistic scheduling can result in significant system-level performance gains and that almost all of these gains can be realized using the polynomial-complexity Whittle’s index policy.

In this paper, we study a family of robust power allocation techniques for multi-input multi-output communication systems with incomplete channel state information at the transmitter side. Precisely, we focus our attention to the case where the transmitter has access only to the modulus of the channel coefficients. We present the robust techniques under two main approaches: the Bayesian (or stochastic) point of view, and the maximin (or worst-case) criterion. The formulation developed in this paper holds for flat fading channels, and also for frequency selective channels; in this latter case, through a multicarrier perspective.