Risk-Averse Anticipation for Dynamic Vehicle Routing

Stochastic and dynamic vehicle routing problems gain increasing attention in the research community. In these problems, routing plans are dynamically updated based on realizations of stochastic information. Due to the complexity of the corresponding Markov decision processes (MDPs), the calculation of optimal policies for these problems is usually not possible and researchers draw on heuristical methods of approximate dynamic programming (ADP). These methods use simulation to approximate the value of a state and decision in the MDP. The simulations are either conducted offline or online. Offline methods such as value function approximations (VFAs) generally neglect the full detail of the state space due to aggregation. Online methods such as rollout algorithms (RAs) are often not able to capture decision and transition space sufficiently due to runtime limitations. In this paper, we alleviate this tradeoff by combining two methods of ADP, an online RA and an offline VFA in two ways. In addition to the integration of the VFA as a base policy into the online RA to strengthen the RA’s simulations, we also limit the RA’s simulation horizon, estimating the remaining reward-to-go again via the VFA. For two stochastic dynamic routing problems from the literature, we show how this combination outperforms state-of-the-art solutions while simultaneously reducing the required time for online calculations.

An approximate dynamic programming (ADP) method has shown good performance in solving optimal control problems in many small‐scale process control applications. The offline computational procedure of ADP constructs an approximation of the optimal “cost‐to‐go” function, which parameterizes the optimal control policy with respect to the state variable. With the approximate “cost‐to‐go” function computed, a multistage optimization problem that needs to be solved online at every sample time can be reduced to a single‐stage optimization, thereby significantly lessening the real‐time computational load. Moreover, stochastic uncertainties can be addressed relatively easily within this framework. Nonetheless, the existing ADP method requires excessive offline computation when applied to a high‐dimensional system. A case study of a reactor and a distillation column with recycle was used to illustrate this issue. Then, several ways were proposed to reduce the computational load so that the ADP method can be applied to high‐dimensional integrated plants. The results showed that the approach is much more superior to NMPC in both deterministic and stochastic cases. © 2009 American Institute of Chemical Engineers AIChE J, 2009

Dynamic vehicle routing problem (DVRP) with single depot has received increasing interest from engineers and scientists. Dynamic multi-depot vehicle routing problem (DMDVRP), an extension of DVRP, however, has not received much attention. In our paper, a distance-based clustering approach is introduced to simplify the DMDVRP by allocating each customer to its nearest depot. Thus, DMDVRP is decomposed to a sequence of DVRPs. An improved ant colony optimisation (IACO) with ant-weight strategy and mutation operation is presented to optimise vehicle routing problem (VRP) in this paper. Moreover, to satisfy the real-time feature of DMDVRP, the nearest addition approach is used to handle the new orders occurring during a time slice on the basis of VRP solution. Finally, the computational results for 17 benchmark problems are reported to validate that IACO with the distance-based clustering approach is more suitable for solving DMDVRP.

Value function approximation for dynamic multi-period vehicle routing

Research on dynamic and stochastic vehicle routing problems received increasing interest in the last decade. It considers a novel problem class, aiming at an appropriate handling of dynamic events combined with the incorporation of stochastic information about possible future events. This survey summarises the recent literature in this area. Besides, the classification according to the available stochastic information, a new classification based on the point in time where substantial computational effort for determining decisions or decision policies arises, is introduced. Furthermore, the difference in solution quality is analysed between approaches which consider either purely dynamic or stochastic problems compared to those which consider both, stochastic and dynamic aspects. A graphical representation demonstrates the strength of the reviewed approaches incorporating dynamic and stochastic information. The survey also gives an overview on the intensity of research for the different problem classes and its benefit in recent years. Finally, guidelines and promising directions for further research are presented.

Chapter 22 Duality Theory and Approximate Dynamic Programming for Pricing American Options and Portfolio Optimization

The Vehicle Routing Problem (VRP) is a complex and high-level set of routing problems. Two of its important variants are the Dynamic Vehicle Routing Problem (DVRP) and the Green Vehicle Routing Problem (GVRP). The first one has become a challenging research topic in the last two decades, in which not all informations are known in advance, but are revealed as the system progresses. The second one is seen as a new application and solution in new logistics patterns, more specifically, for finding routes of vehicles to serve a set of customers while minimizing the total amount of CO 2 emissions, by increasing the loading rate and reducing the number of empty trips could reduce from 10% to 40% km traveled and therefore CO 2 emissions [1]. In this paper, we combine these two variants (DVRP and GVRP), where we try to minimize, in the dynamic environment, the greenhouse gas especially the carbon dioxide CO 2 , which is the immediate consequence of the depletion of the ozone layer and time duration. We will call this combination the Dynamic Green Vehicle Routing Problem (DGVRP). To study this new problem, we present the technique employed to estimate the amount of CO 2 emissions, the emission matrix and integrate them into the DGVRP model, then we propose the hybridization based on Ant Colony Optimization (ACO) algorithm with a Large Neighborhood Search (LNS) algorithm to solve our problem. The effectiveness of this approach is tested on a set of the dynamic green problems instances, which are adopted in this work from the static GVRP benchmark datasets.

Recent advances in commercial vehicle operations (CVO), especially in communication and information technologies, allow the study of dynamic vehicle routing problems under new and updated information, such as traffic conditions and new customers. Two major operational benefits of CVO include ( a) dynamically assigning vehicles to time-sensitive demands, and ( b) efficiently rerouting vehicles according to current traffic conditions. In this research, stochastic vehicle routing problems (SVRP) are considered and extended to incorporate real-time information for dynamic vehicle routing problems. The SVRP model is formulated by a chance-constrained model and is solved by CPLEX with branch-and-bound techniques. Numerical experiments are conducted in a Taichung city network to investigate dynamic vehicle routing strategies under real-time information supply strategies and to assess the effectiveness of such strategies in a dynamic perspective.

Within the wide scope of logistics management,transportation plays a central role and is a crucialactivity in both production and service industry.Among others, it allows for the timely distributionof goods and services between suppliers, productionunits, warehouses, retailers, and final customers.More specifically, Vehicle Routing Problems(VRPs) deal with the design of a set of minimal costroutes that serve the demand for goods orservices of a set of geographically spread customers,satisfying a group of operational constraints.While it was traditionally a static problem, recenttechnological advances provide organizations withthe right tools to manage their vehicle fleet in realtime. Nonetheless, these new technologies alsointroduce more complexity in fleet managementtasks, unveiling the need for decision support systemsdedicated to dynamic vehicle routing. In thiscontext, the contributions of this Ph.D. thesis arethreefold : (i) it presents a comprehensive reviewof the literature on dynamic vehicle routing ; (ii)it introduces flexible optimization frameworks thatcan cope with a wide variety of dynamic vehiclerouting problems ; (iii) it defines a new vehicle routingproblem with numerous applications.

Referring to the optimal tracking guidance of aircraft, the conventional time based kinematics model is transformed into a downrange based model by independent variable replacement. The deviations of in-flight altitude and flight path angle are penalized and corrected to achieve high precision tracking of reference trajectory. The tracking problem is solved as a linear quadratic regulator applying small perturbation theory, and the approximate dynamic programming method is used to cope with the solving of finite-horizon optimization. An actor-critic structure is established to approximate the optimal tracking controller and minimum cost function. The least squares method and Adam optimization algorithm are adopted to learn the parameters of critic network and actor network, respectively. A boosting trajectory with maximum final velocity is generated by Gauss pseudospectral method for the validation of guidance strategy. The results show that the trained feedback control parameters can effectively resist random wind disturbance, correct the initial altitude and flight path angle deviations, and achieve the goal of following a given trajectory.

This work presents an adaptive optimal control algorithm based integral sliding mode control law for a class of continuous-time systems with input disturbance or uncertain and unknown parameters. The main objective is to find a general form of integral sliding mode control law can assure that the system states are forced to reach a sliding surface in the initial finite time. Then, an adaptive optimal control based on the approximate dynamic programming method is responsible for the robust stability of the closed-loop system. Finally, the theoretical analysis and simulation results demonstrate the performance of the proposed algorithm for a wheel inverted pendulum system.

In the past decade, there has been considerable research interest in learning control methods based on reinforcement learning (RL) and approximate dynamic programming (ADP). As an important class of function approximation techniques, kernel methods have been recently applied to improve the generalization ability of RL and ADP methods but most previous works were only based on simulation. This paper focuses on experimental studies of real-time online learning control for nonlinear systems using kernel-based ADP methods. Specifically, the kernel-based dual heuristic programming (KDHP) method is applied and tested on real-time control systems. Two kernel-based online learning control schemes are presented for uncertain nonlinear systems by using simulation data and online sampling data, respectively. Learning control experiments were performed on a single-link inverted pendulum system as well as a double-link inverted pendulum system. From the experimental results, it is shown that both online learning control schemes, either using simulation data or using real sampling data, are effective for approximating near-optimal control policies of nonlinear dynamical systems with model uncertainties. In addition, it is demonstrated that KDHP can achieve better performance than conventional DHP, which uses multilayer perceptron neural networks.

Unit commitment (UC) is crucial for reducing the generation cost of short-term power system scheduling. This study proposes a new approximate dynamic programming (ADP) method to solve large-scale UC considering ramp rate limits of thermal units. To solve the “curse of dimensionality,” a policy iteration algorithm using post-decision state variables is utilized for value function approximation. The post-decision state variables dramatically simplify the algorithm by avoiding the need to compute the value of all feasible states within Bellman’s equation. The proposed method is applied to systems ranging in size from 10 to 1000 units to optimize the generation scheduling of the units in a day. And the time horizon of the UC problem is split into 96 time periods. It is shown that ADP can converge to promising results in polynomial time complexity for large-scale UC.

Learning-based robust output tracking control for unknown discrete-time nonlinear systems with dynamic uncertainty

There is a growing interest in using electric vehicles (EVs) and drones for many applications. However, battery-oriented issues, including range anxiety and battery degradation, impede adoption. Battery swap stations are one alternative to reduce these concerns that allow the swap of depleted for full batteries in minutes. We consider the problem of deriving actions at a battery swap station when explicitly considering the uncertain arrival of swap demand, battery degradation, and replacement. We model the operations at a battery swap station using a finite horizon Markov decision process model for the stochastic scheduling, allocation, and inventory replenishment problem (SAIRP), which determines when and how many batteries are charged, discharged, and replaced over time. We present theoretical proofs for the monotonicity of the value function and monotone structure of an optimal policy for special SAIRP cases. Because of the curses of dimensionality, we develop a new monotone approximate dynamic programming (ADP) method, which intelligently initializes a value function approximation using regression. In computational tests, we demonstrate the superior performance of the new regression-based monotone ADP method compared with exact methods and other monotone ADP methods. Furthermore, with the tests, we deduce policy insights for drone swap stations.