Regular Ordering and Applications in Control Policies

Stochastic Event Graph (SEG) is a useful tool to model some discrete event dynamic systems and evaluate their performance. In this paper, the evolution of the system is analysed by unfolded SEG and its state equations are presented. By using the tree structure of the correlative property of the state variables, an algorithm for the probability distributions of state variables is derived. Finally, for an example of decision making organization, the steady solutions are reached after two iterations. In comparison with Markov Chain analysis, this algorithm has better convergence, and can solve problems with general distributions of service time. It can obtain not only steady distributions, but also transient properties. >

We present an emulation-based controller and transmission policy design procedure for nonlinear wireless networked control systems. The objective is to ensure the stability of the closed-loop system, in a stochastic sense, together with given control performance, while minimizing the average power used for communications. The controller is designed by emulation, i.e., ignoring the network, and the transmission power is given by threshold policies. These policies involve waiting a given amount of time since the last successful transmission instant, as well as requiring that the measured wireless channel gain is above a given threshold, before attempting a new transmission. Two power control laws are investigated: 1) a constant power and 2) a power level inversely proportional to the channel gain. We explain how to select the waiting time, the channel threshold, and the power level to minimize the induced average communication power while ensuring the desired control objectives.

Summary Uncertainty is a major challenge in reservoir management. To take the uncertainty into consideration, optimization can be carried out over a set of scenarios. Most approaches on reservoir management under uncertainty optimize a sequence of control inputs applied to all scenarios over the prediction horizon; hence, they are open-loop predictions. In this paper, we optimize over control policies, as opposed to a sequence of control inputs, to obtain closed-loop predictions. The policies are specified as a set of implicit algebraic equations, allowing for efficient gradient calculation by an adjoint simulation. The method is compared with the more traditional open-loop approach in a case study, indicating a significant potential for reservoir optimization by use of closed-loop predictions.

We consider a linear control system, where the sensors communicate to the controller via a wireless network in which the probability of successful data transmissions is an increasing function of the selected radio signal power. We tackle the problem of jointly selecting the transmission power at every transmission time as a function of the state, much like in event-triggered control systems, and of designing a controller in order to minimize an average quadratic cost. Our proposed power scheduler and corresponding control policy are shown to outperform the optimal control policy when the radio signal power is constant at all times, while using at most the same average transmission power. For this result to hold, the function relating the transmission power to the probability of successful transmission must be convex in the region of interest. We call such a policy LQ-power consistent.

In this paper, the downlink file transmission within a finite lifetime is optimized with the assistance of wireless cache nodes. Specifically, the number of requests within the lifetime of one file is modeled as a Poisson point process. The base station multicasts files to downlink users and the selected the cache nodes, so that the cache nodes can help to forward the files in the next file request. Thus we formulate the downlink transmission as a Markov decision process (MDP) with random number of stages, where transmission power and time on each transmission are the control policy. Due to random number of file transmissions, we first proposed a revised Bellman's equation, where the optimal control policy can be derived. In order to address the prohibitively huge state space, we also introduce a low-complexity sub-optimal solution based on a linear approximation of the value function. The approximated value function can be calculated analytically, so that conventional numerical value iteration can be eliminated. Moreover, the gap between the approximated value function and the real value function is bounded analytically. It is shown by simulation that, with the approximated MDP approach, the proposed algorithm can significantly reduce the resource consumption at the base station.

A multiplicative regularization scheme with edge-preserving characteristics is applied to the inversion of electrical impedance tomography (EIT) data. This scheme employs a multiplicative cost function of a weighted L2-norm regularization function and the data misfit function. It avoids the use of a weighting factor when the regularization term is added to the cost function and allows an adaptive weighting between data misfit and the regularization function. Gauss-Newton method is used to minimize the multiplicative cost function. In this work, we extend the weighted L2-norm regularization scheme onto a triangular grid with an updated formula for gradient and divergence operators. This scheme is tested using synthetic data. The reconstructed images show good piecewise constant characteristics and noise-resistance performance.

The prevention and control of COVID-19 in megacities is under large pressure because of tens of millions and high-density populations. The majority of epidemic prevention and control policies implemented focused on travel restrictions, which severely affected urban mobility during the epidemic. Considering the impacts of epidemic and associated control policies, this study analyzes the relationship between COVID-19, travel of residents, Point of Interest (POI), and social activities from the perspective of taxi travel. First, changes in the characteristics of taxi trips at different periods were analyzed. Next, the relationship between POIs and taxi travels was established by the Geographic Information System (GIS) method, and the spatial lag model (SLM) was introduced to explore the changes in taxi travel driving force. Then, a social activities recovery level evaluation model was proposed based on the taxi travel datasets to evaluate the recovery of social activities. The results demonstrated that the number of taxi trips dropped sharply, and the travel speed, travel time, and spatial distribution of taxi trips had been significantly influenced during the epidemic period. The spatial correlation between taxi trips was gradually weakened after the outbreak of the epidemic, and the consumption travel demand of people significantly decreased while the travel demand for community life increased dramatically. The evaluation score of social activity is increased from 8.12 to 74.43 during the post-epidemic period, which may take 3–6 months to be fully recovered as a normal period. Results and models proposed in this study may provide references for the optimization of epidemic control policies and recovery of public transport in megacities during the post-epidemic period.

The Acrobot is an underactuated robot and provides challenging control problems due to its nonlinearity. We try creating global control policies depending on the limitation of memory that is available for recording them. A swinging up task of the Acrobot is solved by dynamic programming (DP) without heuristics. The result of DP is compressed to an arbitrary size of control policy by our vector quantization (VQ) method. A chaotic but performable control policy can be obtained in the global state space of the Acrobot by DP. Moreover, we can verify that our VQ algorithm can reduce the dimension of the control policy from four to two in the process of compression.

In solving optimal control problems numerically, the control policy is parametrized over a finite number of time intervals. The choice of the number of such time stages and the type of parametrization can be very important in establishing the nature of the optimal control policy. Several examples are used to show the importance of parametrization. For some systems, a highly oscillatory control policy improves the performance index. Such control policies can be readily established with iterative dynamic programming by the use of a large number of time stages of flexible length. In the optimization of a continuous stirred tank reactor, it is found that when the feed rates are periodically switched from lower to upper bounds, the profit function is improved beyond a smoother operation, which was previously believed to be the optimal operating policy. †Dedicated to Prof. N.U. Ahmed on the occasion of his 70th birthday.

In this paper, we tackle the problem of selecting a sparse data transmission sequence for a networked control system while guaranteeing a certain ${\mathcal{L}_2}$-induced norm bound with respect to an exogenous disturbance input. As the main contribution of this work, for every periodic transmission sequence, we provide a transmission sequence counterpart, aperiodic in general, that results in fewer or at most the same number of transmissions while still guaranteeing an ${\mathcal{L}_2}$ control policy with the same ${\mathcal{L}_2}$-induced norm bound as that of the periodic transmission policy. In this sense, the proposed transmission sequence is called ${\mathcal{L}_2}$-consistent. Moreover, we show that as the horizon approaches infinity, the proposed ${\mathcal{L}_2}$consistent transmission sequence counterpart of every periodic transmission sequence approaches a periodic transmission sequence with an equal or larger time period.

Dynamic production control in parallel processing systems under process queue time constraints

This paper focuses on the data-driven controller design for optimal control problems of nonlinear nonaffine discrete-time systems. A novel policy gradient and Q-learning (PGQL) adaptive algorithm which learns the optimal control policy from real empirical data is developed without requiring system dynamics. A policy iteration scheme is designed to iteratively update the approximate Q-function, and the control policy is improved via gradient method until they converge to the bounded regions of the optimal Q-function and the optimal control policy, respectively. Two neural networks (NNs) are employed to realize the developed algorithm. Moreover, the convergence analysis of approximate Q-function is established. Since the control policy is parameterized, it can be upgraded through updating the actor-NN parameters in the direction of the performance gradient. Finally, the simulation results are given to verify the performance of the developed PGQL adaptive algorithm.

Part I of the thesis investigates novel urban traffic state estimation methods utilizing probe vehicle data. Chapter 2 proposes a method to integrate the collective effect of dispersed probe data with traffic kinematic wave theory and data mining techniques to model the spatial and temporal dynamics of queue formation and dissipation in arterials. The queue estimation method captures interdependencies in queue evolutions of successive intersections, and moreover, the method is applicable in oversaturated conditions and includes a queue spillover statistical inference procedure. Chapter 3 develops a travel time reliability model to estimate arterial route travel times distribution (TTD) considering spatial and temporal correlations between traffic states in consecutive links. The model uses link-level travel time data and a heuristic grid clustering method to estimate the state structure and transition probabilities of a Markov chain. By applying the Markov chain procedure, the correlation between states of successive links is integrated and the route-level TTD is estimated. The methods in Part I are tested with various probe vehicle penetration rates on case studies with field measurements and simulated data. The methods are straightforward in implementation and have demonstrated promising performance and accuracy through numerous experiments. Part II studies network-level modeling and control of large-scale urban networks. Chapter 4 is the pioneer that introduces the urban perimeter control for two-region urban cities as an elegant control strategy to decrease delays in urban networks. Perimeter controllers operate on the border between the two regions, and manipulate the percentages of transfer flows between the two regions, such that the number of trips reaching their destinations is maximized. The optimal perimeter control problem is solved by the model predictive control (MPC) scheme, where the prediction model and the plant (reality) are formulated by macroscopic fundamental diagrams (MFD). Chapter 5 extends the perimeter control strategy and MFD modeling to mixed urban-freeway networks to provide a holistic approach for large-scale integrated corridor management (ICM). The network consists of two urban regions, each one with a well-defined MFD, and a freeway, modeled by the asymmetric cell transmission model, that is an alternative commuting route which has one on-ramp and one off-ramp within each urban region. The optimal traffic control problem is solved by the MPC approach to minimize total delay in the entire network considering several control policies with different levels of urban-freeway control coordination. Chapter 6 integrates traffic heterogeneity dynamics in large-scale urban modeling and control to develop a hierarchical control strategy for heterogeneously congested cities. Two aggregated models, region- and subregion-based MFDs, are introduced to study the effect of link density heterogeneity on the scatter and hysteresis of MFD. A hierarchical perimeter flow control problem is proposed to minimize the network delay and to homogenize the distribution of congestion. The first level of the hierarchical control problem is solved by the MPC approach, where the prediction model is the aggregated parsimonious region-based MFD and the plant is the subregion-based MFD, which is a more detailed model. At the lower level, a feedback controller tries to maximize the network outflow, by increasing regional homogeneity.

We study a model for the exploitation of renewable stocks developed in [5]. In this particular control problem, the control law contains a measurable and an impulsive control component. We formulate Pontryagin's maximum principle for this kind of control problems (see [3]), proving first order necessary conditions of optimality. Manipulating the correspondent Lagrange multipliers we manage to define two special switch functions, that allow the complete description the optimal trajectories and control policies, for all possible initial conditions in the phase plane.

In this paper, we consider a class of continuous-time, continuous-space stochastic optimal control problems. Using the Markov chain approximation method and recent advances in sampling-based algorithms for deterministic path planning, we propose a novel algorithm called the incremental Markov Decision Process to incrementally compute control policies that approximate arbitrarily well an optimal policy in terms of the expected cost. The main idea behind the algorithm is to generate a sequence of finite discretizations of the original problem through random sampling of the state space. At each iteration, the discretized problem is a Markov Decision Process that serves as an incrementally refined model of the original problem. We show that with probability one, (i) the sequence of the optimal value functions for each of the discretized problems converges uniformly to the optimal value function of the original stochastic optimal control problem, and (ii) the original optimal value function can be computed efficiently in an incremental manner using asynchronous value iterations. Thus, the proposed algorithm provides an anytime approach to the computation of optimal control policies of the continuous problem. The effectiveness of the proposed approach is demonstrated on motion planning and control problems in cluttered environments in the presence of process noise.