Reinforcement learning approach for resource allocation in humanitarian logistics

ABSTRACTLarge-scale disasters and catastrophic events typically result in a significant shortage of critical resources, posing a great challenge to allocating limited resources among different affected areas to improve the quality of emergency logistics operations. This article pays attention to the performance of resource allocation, which includes three metrics: efficiency, effectiveness, and equity, respectively corresponding to economic cost, service quality, and fairness. In particular, the effectiveness metric considers human suffering by depicting it as deprivation cost, an economic valuation measurement that has been recently proposed and the equity metric concerns about the service equality at the end of planning horizon. A nonlinear integer model is first proposed and then an equivalent dynamic programming model is developed to avoid the nonlinear terms created by the introduction of the deprivation cost. The dynamic programming method can solve small-scale problems to optimality but meets difficulty when solving medium- and large-scale problems, due to the curse of dimensionality. Therefore, an approximate dynamic programming algorithm, called the rollout algorithm, is proposed to overcome this computational difficulty. The computational complexity of the proposed algorithm is theoretically analyzed. Furthermore, a modified version of the rollout algorithm is presented, with its computational complexity analyzed. Extensive numerical experiments are conducted to test the performance of the proposed algorithms, and the experimental results demonstrate that the initially proposed rollout algorithm yields optimal or near-optimal solutions within a reasonable amount of time. In addition, the impacts of some important parameters are investigated and managerial insights are drawn.

The air-rail inter-modal transport is a feasible choice to enlarge the freight service scope of high-speed railway. Essentially, optimising the service plan for high-speed rail express under inter-modal mainly involves determining the train and flight trips, and space-time route selections for each batch of express shipment from the origin to the destination. We construct an extended space-time network to capture the transport and transfer space-time attributes of the serviced express shipments. A multi-commodity flow model is then established with a series of practical constraints. The Lagrangian relaxation algorithm is designed to decompose the original problem into a shortest path problem of a single-batch express shipment in a multi-dimensional network. The sub-problem is solved by the dynamic programming method, and a heuristic algorithm based on sub-gradient sorting is designed to ensure the feasibility of the dual solutions. In order to compare the performance of the traditional solver method with that of the LR algorithm, a nonlinear mixed-integer programming model was constructed in the appendix and solved by using the DICOPT solver. Taking the Shanghai-Kunming corridor as an example, the experimental results demonstrate that the LR algorithm can obtain high-quality solutions within a relatively short time, while the traditional solver method has certain limitations. Furthermore, the inter-modal transport is validated with a significant advantage in expanding the service scope of express demand. The research results are of great theoretical significance for the rational allocation of transportation resources and enhancement of the quality and efficiency of high-speed rail express services.

Optimal load clipping with time of use rates

Mobile ad hoc networks (MANETs) consist of a set of nodes which can move freely and communicate with each other wirelessly. Due to the movement of nodes and unlike wired networks, the available routes used among the nodes for transmitting data packets are not stable. Hence, proposing real-time routing protocols for MANETs is regarded as one of the major challenges in this research domain. Algorithms compatible with the changes created in the network due to the nodes' movements are of high significance. For reducing data packet transmission time among nodes, not only should route shortness be considered but also route stability should be taken into consideration. Since available factors in different environments have specific behavior patterns especially in human environments, the parameters of link stability and route shortness were taken into consideration and the reinforcement learning was used to propose a method so as to make the best choice among the neighbors at any moment to transmit a packet to the destination. That is, the proposed method was aimed at predicting the behavior pattern of the nodes in relation to the target node through using reinforcement learning. The proposed method used Q-learning algorithm which has more homogeneity to estimate the value of actions. Simulation results in OPNET demonstrate the superiority of the proposed scheme over conventional MANET routing methods.

In this paper, a stochastic model predictive control (MPC) method based on reinforcement learning is proposed for energy management of plug-in hybrid electric vehicles (PHEVs). Firstly, the power transfer of each component in a power-split PHEV is described in detail. Then an effective and convergent reinforcement learning controller is trained by the Q-learning algorithm according to the driving power distribution under multiple driving cycles. By constructing a multi-step Markov velocity prediction model, the reinforcement learning controller is embedded into the stochastic MPC controller to determine the optimal battery power in predicted time domain. Numerical simulation results verify that the proposed method achieves superior fuel economy that is close to that by stochastic dynamic programming method. In addition, the effective state of charge tracking in terms of different reference trajectories highlight that the proposed method is effective for online application requiring a fast calculation speed.

Currently many non-tractable considered problems have been solved satisfactorily through methods of approximate optimization called metaheuristic. These methods use non-deterministic approaches that find good solutions which, however, do not guarantee the determination of the global optimum. The success of a metaheuristic is conditioned by capacity to adequately alternate between exploration and exploitation of the solution space. A way to guide such algorithms while searching for better solutions is supplying them with more knowledge of the solution space (environment of the problem). This can to be made in terms of a mapping of such environment in states and actions using Reinforcement Learning. This paper proposes the use of a technique of Reinforcement Learning - Q-Learning Algorithm - for the constructive phase of GRASP and Reactive GRASP metaheuristic. The proposed methods will be applied to the symmetrical traveling salesman problem.

Contemporary behavioral learning theory provides a comprehensive description of how we and other animals learn, and places behavioral flexibility and automaticity at heart of adaptive behaviors. However, to our knowledge, the computations supporting the interactions between deliberative and habitual decision-making systems are still poorly understood. Previous functional magnetic resonance imaging (fMRI) results suggest that the dorsal striatum host complementary computations that may differentially support deliberative and habitual processes [1] in the form of a dynamical interplay rather than a serial recruitment of strategies. From the same instrumental task, we develop a dual-system computational model of the two systems that can predict both performance (i.e., participant choices) and modulations in reaction times during learning. The instrumental task is a trial-and-error learning task requiring participants to find the correct associations between color stimuli and finger responses. To model the habitual system, we use a simple Q-learning algorithm (QL) [2] whose properties are fast responses, but slow convergence. For the deliberative (i.e goal-directed) system, we propose a new Bayesian Working Memory (BWM) which searches for information in the history of previous trials and stops as soon as the uncertainty on the action to perform decreases below a certain threshold. Last, we also propose a model for QL and BWM coordination. Currently, most models of system selection tend to control action selection concurrently, using either the deliberative or habitual model according to uncertainty criteria [3,4]. Only one model has investigated the relation between working memory and reinforcement learning [5] without, however explicitly modeling the temporal aspect of memory manipulation. In our approach, we propose a model for QL and BWM coordination. QL and BWM are merged such that the expensive memory manipulation is under control of, among others, the level of convergence of the habitual learning. Consequently, we also predict specific reaction times for each model that can be compared with the evolution of reaction times in instrumental learning tasks. Models are optimized for each subject with the NSGA-2 multi-objective evolutionary algorithm. The first fitness function is the Bayesian Information Criterion for individual choices. The second fitness function is also a likelihood that maximizes the probability of performing reaction times similar to humans. We compare the ability of the new model to explain human behavior with the QL or BWM only, as well as with a combination of these models based on [4], which reveals that the proposed model is in general more accurate. To conclude, we suggest that a close combination of BWM and QL better explains both choices and reaction times for most participants.

Relief network design for predictable disasters is a key issue in humanitarian logistics. However, existing studies on relief network design have not simultaneously considered multiple relief decision stages and the uncertainties which are reduced by improved forecast accuracy as a disaster approaches. These aspects are critical for efficient relief activities. The present study investigates a new relief network design problem for predictable disasters, especially typhoons, which have become increasingly frequent and severe in recent years. It integrates facility location decisions before any specific disaster warnings are issued, relief supply deployment and evacuation decisions between a warning and the onset of the disaster, and relief supply distribution decisions after a disaster strikes. This study also considers time-dependent uncertainties of the disaster’s trajectory and intensity together. For the problem, a novel three-stage hybrid distributionally robust and robust optimization (3HDRO) model is proposed. To make it computationally tractable, the 3HDRO model is transformed into a deterministic equivalent (DE) model. A scenario-based decomposition heuristic algorithm is then designed to solve the DE model for large-scale instances. A case study of historical typhoons in Guangdong Province, China, is used to gain insights into the critical model parameters for disaster relief activities. Furthermore, experimental results on randomly generated instances demonstrate the effectiveness and efficiency of the proposed model and algorithm.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Device-to-Device (D2D) communication is a promising technology that can reduce the burden on cellular networks while increasing network capacity. In this paper, we focus on the channel resource allocation and power control to improve the system resource utilization and network throughput. Firstly, we treat each D2D pair as an independent agent. Each agent makes decisions based on the local channel states information observed by itself. The multi-agent Reinforcement Learning (RL) algorithm is proposed for our multi-user system. We assume that the D2D pair do not possess any information on the availability and quality of the resource block to be selected, so the problem is modeled as a stochastic non-cooperative game. Hence, each agent becomes a player and they make decisions together to achieve global optimization. Thereby, the multi-agent Q-learning algorithm based on game theory is established. Secondly, in order to accelerate the convergence rate of multi-agent Q-learning, we consider a power allocation strategy based on Fuzzy Cmeans (FCM) algorithm. The strategy firstly groups the D2D users by FCM, and treats each group as an agent, and then performs multi-agent Q-learning algorithm to determine the power for each group of D2D users. The simulation results show that the Q-learning algorithm based on multi-agent can improve the throughput of the system. In particular, FCM can greatly speed up the convergence of the multi-agent Q-learning algorithm while improving system throughput.

The Internet of Everything (IoE) is accelerating the demand for intelligent, low-latency, and highly reliable communication systems to support automation and real-time decision-making. In dense urban and industrial environments, unmanned aerial vehicles (UAVs) are increasingly utilized to extend network coverage, improve connectivity, and enable dynamic data collection. However, managing RIS-assisted UAV-enabled IoE networks poses significant challenges, including accurate signal prediction, high computational complexity, and decentralized task assignment. To address these issues, we propose a novel RIS-empowered UAV-based Dynamic Area of Coverage (DAC) architecture. In this framework, UAVs equipped with reconfigurable intelligent surfaces (RIS) adaptively adjust the phase of reflected signals to optimize wireless channel conditions, suppress interference, and enhance signal quality.We formulate the Dynamic Area of Coverage with Location, Resource Allocation, and Trajectory Optimization (DAC-LRT) problem as a mixed-integer nonlinear programming (MINLP) model, aiming to jointly optimize UAV positioning, power distribution, and trajectory control to maximize real-time downlink capacity and ensure energy efficiency. To solve the DAC-LRT problem in dynamic and large-scale IoE environments, we design a Multi-Agent Distributed Deep Deterministic Policy Gradient (MAD3PG) algorithm. MAD3PG enables decentralized and adaptive policy learning by allowing UAVs to derive optimal actions directly from environmental observations. Simulation results demonstrate that our proposed approach significantly outperforms state-of-the-art methods, achieving improvements of 82.92%, 78.02%, and 71.9% in downlink capacity, coverage ratio, average throughput, and spectral efficiency over Deep Deterministic Policy Gradient (DDPG), Asynchronous Advantage Actor-Critic (A3C), and Greedy algorithms, respectively.

Non-orthogonal multiple access (NOMA) is a key technology for future wireless networks, enabling improved spectral efficiency and massive device connectivity. However, optimizing power allocation, subchannel assignment, and cell association in multi-cell uplink NOMA systems is challenging due to user mobility and the NP-hard nature of the problem. This paper addresses these challenges by formulating the problem as a mixed-integer non-linear programming (MINLP) model to maximize energy efficiency (EE). We propose a deep reinforcement learning framework that employs deep Q-networks (DQN) for cell association and subchannel assignment, and twin delayed deep deterministic policy gradient (TD3) for power allocation. Simulation results reveal significant EE improvements, with multi-agent TD3 (MATD3) outperforming traditional Lagrange methods and multi-agent deep deterministic policy gradient (MADDPG). Furthermore, the proposed method exhibits robust adaptability to user mobility and superior performance in multi-cell environments, effectively mitigating inter-cell interference and enhancing resource allocation in dynamic scenarios.

Disasters were unpredictable and unavoidable, necessitating adequate disaster response plans. Humanitarian logistics, which gained prominence after the 2004 Asian tsunami, played an important part in crisis prevention. Despite weak links in relief chain management, issues remained. Humanitarian logistics had a huge impact on disaster relief efforts, as studies revealed a variety of challenges, including a lack of sufficient SCM software and poor facilities. Despite its importance, few studies have explored the impact of disaster assistance on humanitarian organizations, particularly in Kenya. The study sought to address this gap by evaluating the impact of humanitarian logistics on emergency response in Nairobi, Kenya. Humanitarian groups, policymakers, and NGOs profited from the research, which informed strategic planning and policy creation to strengthen disaster response operations. The Resource Dependency Theory (RDT) proposed that organizations rely on external resources for survival and growth. It underlined the importance of forming strategic alliances to secure resources. In humanitarian settings, logistics were critical in procuring and allocating resources for successful emergency response. The research methodology used a cross-sectional descriptive design to investigate the association between humanitarian logistics management techniques and emergency response in Kenyan humanitarian organizations. Semi-structured questionnaires were used to obtain data from 64 Nairobi-based firms. Data analysis required the use of both descriptive and inferential statistical techniques, such as regression and correlation analysis. The study found a significant positive association (β = 0.732, p < 0.001) between logistics management practices and emergency response, indicating that improving logistics leads to better response outcomes. The study recommended prioritizing logistical development and resource allocation to improve emergency response capabilities.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

The applicability of the dynamic programming method to two-dimensional slope stability analyses is studied. The critical slip surface is defined as the slip surface that yields the minimum value of an optimal function. The only assumption regarding the shape of the critical slip surface is that the surface is an assemblage of linear segments. Stresses acting along the critical slip surface are computed using a finite element stress analysis. Assumptions associated with limit equilibrium methods of slices related to the shape of the critical slip surface and the relationship between interslice forces are no longer required. A computer program named DYNPROG was developed based on the proposed analytical procedure, and numerous example problems have been analyzed. Results obtained when using DYNPROG were compared with those obtained when using several well-known limit equilibrium methods. The comparisons demonstrate that the dynamic programming method provides a superior solution when compared with conventional limit equilibrium methods. Analyses conducted also show that factors of safety computed when using the dynamic programming method are generally slightly lower than those computed using conventional limit equilibrium methods of slices; however, as Poisson's ratio approaches 0.5, the computed factors of safety from the dynamic programming method and the limit equilibrium method appear to become similar.Key words: dynamic programming, slope stability, stress analysis, optimization theory, limit equilibrium methods of slices.