Optimizing inventory management through reinforcement learning

Problem definition: The beer game is widely used in supply chain management classes to demonstrate the bullwhip effect and the importance of supply chain coordination. The game is a decentralized, multiagent, cooperative problem that can be modeled as a serial supply chain network in which agents choose order quantities while cooperatively attempting to minimize the network’s total cost, although each agent only observes local information. Academic/practical relevance: Under some conditions, a base-stock replenishment policy is optimal. However, in a decentralized supply chain in which some agents act irrationally, there is no known optimal policy for an agent wishing to act optimally. Methodology: We propose a deep reinforcement learning (RL) algorithm to play the beer game. Our algorithm makes no assumptions about costs or other settings. As with any deep RL algorithm, training is computationally intensive, but once trained, the algorithm executes in real time. We propose a transfer-learning approach so that training performed for one agent can be adapted quickly for other agents and settings. Results: When playing with teammates who follow a base-stock policy, our algorithm obtains near-optimal order quantities. More important, it performs significantly better than a base-stock policy when other agents use a more realistic model of human ordering behavior. We observe similar results using a real-world data set. Sensitivity analysis shows that a trained model is robust to changes in the cost coefficients. Finally, applying transfer learning reduces the training time by one order of magnitude. Managerial implications: This paper shows how artificial intelligence can be applied to inventory optimization. Our approach can be extended to other supply chain optimization problems, especially those in which supply chain partners act in irrational or unpredictable ways. Our RL agent has been integrated into a new online beer game, which has been played more than 17,000 times by more than 4,000 people.

Decision-making in supply chains is challenged by high complexity, a combination of continuous and discrete processes, integrated and interdependent operations, dynamics, and adaptability. The rapidly increasing data availability, computing power and intelligent algorithms unveil new potentials in adaptive data-driven decision-making. Reinforcement Learning, a class of machine learning algorithms, is one of the data-driven methods. This semi-systematic literature review explores the current state of the art of reinforcement learning in supply chain management (SCM) and proposes a classification framework. The framework classifies academic papers based on supply chain drivers, algorithms, data sources, and industrial sectors. The conducted review revealed a few critical insights. First, the classic Q-learning algorithm is still the most popular one. Second, inventory management is the most common application of reinforcement learning in supply chains, as it is a pivotal element of supply chain synchronisation. Last, most reviewed papers address toy-like SCM problems driven by artificial data. Therefore, shifting to industry-scale problems will be a crucial challenge in the next years. If this shift is successful, the vision of data-driven decision-making in real-time could become a reality.

The objective of this paper is to examine the use and applications of reinforcement learning (RL) techniques in the production planning and control (PPC) field addressing the following PPC areas: facility resource planning, capacity planning, purchase and supply management, production scheduling and inventory management. The main RL characteristics, such as method, context, states, actions, reward and highlights, were analysed. The considered number of agents, applications and RL software tools, specifically, programming language, platforms, application programming interfaces and RL frameworks, among others, were identified, and 181 articles were sreviewed. The results showed that RL was applied mainly to production scheduling problems, followed by purchase and supply management. The most revised RL algorithms were model-free and single-agent and were applied to simplified PPC environments. Nevertheless, their results seem to be promising compared to traditional mathematical programming and heuristics/metaheuristics solution methods, and even more so when they incorporate uncertainty or non-linear properties. Finally, RL value-based approaches are the most widely used, specifically Q-learning and its variants and for deep RL, deep Q-networks. In recent years however, the most widely used approach has been the actor-critic method, such as the advantage actor critic, proximal policy optimisation, deep deterministic policy gradient and trust region policy optimisation.

Estimation bias is an important index for evaluating the performance of reinforcement learning (RL) algorithms. The popular RL algorithms, such as Q -learning and deep Q -network (DQN), often suffer overestimation due to the maximum operation in estimating the maximum expected action values of the next states, while double Q -learning (DQ) and double DQN may fall into underestimation by using a double estimator (DE) to avoid overestimation. To keep the balance between overestimation and underestimation, we propose a novel integrated DE (IDE) architecture by combining the maximum operation and DE operation to estimate the maximum expected action value. Based on IDE, two RL algorithms: 1) integrated DQ (IDQ) and 2) its deep network version, that is, integrated double DQN (IDDQN), are proposed. The main idea of the proposed RL algorithms is that the maximum and DE operations are integrated to eliminate the estimation bias, where one estimator is stochastically used to perform action selection based on the maximum operation, and the convex combination of two estimators is used to carry out action evaluation. We theoretically analyze the reason of estimation bias caused by using nonmaximum operation to estimate the maximum expected value and investigate the possible reasons of underestimation existence in DQ. We also prove the unbiasedness of IDE and convergence of IDQ. Experiments on the grid world and Atari 2600 games indicate that IDQ and IDDQN can reduce or even eliminate estimation bias effectively, enable the learning to be more stable and balanced, and improve the performance effectively.

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

Target searching is crucial in real-world scenarios such as search and rescue in disaster sites and battlefield target reconnaissance. Unmanned aerial vehicles (UAVs) are an ideal technical solution for target searching in large-scale and high-risk areas because they are agile, low cost, and able to collaborate and carry different sensors. In complex scenarios like battlefields, due to the lack of communication infrastructures and the intensive interference, UAVs often operate in communication denied environments. As a result, fast and reliable communication channels between UAVs and ground operators are difficult to establish. Thus, in such conditions, UAVs must be able to complete tasks autonomously and intelligently, without receiving real-time commands from the operators. With the rapid advances in artificial intelligence, reinforcement learning has shown potentiality for solving continuous decision problems. The target searching problem studied in this paper falls into this category and is suitable for adopting reinforcement learning technologies. However, the feasibility of reinforcement learning in UAV-based target searching in communication denied environments is not clear and, thus, requires in-depth investigations. As a pilot study in this direction, this paper models the target searching problem in communication denied and confrontation situations and proposes a simulation environment based on this model. Extensive experiments are conducted to answer the following questions. (1) Can reinforcement learning be applied in target searching by multi-UAVs in communication denied environments? (2) What are the advantages and disadvantages of different reinforcement learning algorithms in solving this problem? (3) How the degree of communication denial influences the performance of these algorithms? The current mainstream reinforcement learning technologies are adopted to perform simulations, whose results are analyzed quantitatively, leading to the following observations. (1) Reinforcement learning can effectively solve target searching problems for multi-UAVs in communication denied environments. (2) Compared with other algorithms, an autonomous decision-making UAV cluster based on a deep Q-network (DQN)exhibits the best problem-solving ability. (3) The algorithm performance changes with the degree of communication denial but remains largely stable when the communication condition varies.

Case-based myopic reinforcement learning for satisfying target service level in supply chain

Active screening is a common approach in controlling the spread of recurring infectious diseases such as tuberculosis and influenza. In this approach, health workers periodically select a subset of population for screening. However, given the limited number of health workers, only a small subset of the population can be visited in any given time period. Given the recurrent nature of the disease and rapid spreading, the goal is to minimize the number of infections over a long time horizon. Active screening can be formalized as a sequential combinatorial optimization over the network of people and their connections. The main computational challenges in this formalization arise from i) the combinatorial nature of the problem, ii) the need of sequential planning and iii) the uncertainties in the infectiousness states of the population. Previous works on active screening fail to scale to large time horizon while fully considering the future effect of current interventions. In this paper, we propose a novel reinforcement learning (RL) approach based on Deep Q-Networks (DQN), with several innovative adaptations that are designed to address the above challenges. First, we use graph convolutional networks (GCNs) to represent the Q-function that exploit the node correlations of the underlying contact network. Second, to avoid solving a combinatorial optimization problem in each time period, we decompose the node set selection as a sub-sequence of decisions, and further design a two-level RL framework that solves the problem in a hierarchical way. Finally, to speed-up the slow convergence of RL which arises from reward sparseness, we incorporate ideas from curriculum learning into our hierarchical RL approach. We evaluate our RL algorithm on several real-world networks. Results show that our RL algorithm can scale up to 10 times the problem size of state-of-the-art (the variant that considers the effect of future interventions but unscalable) in terms of planning time horizon. Meanwhile, it outperforms state-of-the-art (the variant that scales up but does not consider the effect of future interventions) by up to 33% in solution quality.

The flying sidekick traveling salesman problem with stochastic travel time: A reinforcement learning approach

Background Recent pandemics, epidemics and outbreaks continually underscore the critical importance of resilient supply chains in healthcare, particularly in clinical laboratories where timely access to supplies is essential for patient care. Supply chain disruptions, whether due to global crises or localized challenges, can significantly impact laboratory operations, leading to delayed diagnoses, compromised patient outcomes, and increased costs. It is estimated that 70%-80% of a patients’ EMR is clinical laboratory results. Despite this, supply chain resilience in clinical laboratories remains an under-addressed issue and remains a buzzword. Methods To evaluate the cost and importance of supply chain resilience in the clinical laboratory, we conducted a comprehensive review and interviews with 9 Healthcare Supply Chain experts analyzing the impact of supply chain disruptions on laboratory operations and patient care. Data was also collected from a diverse range of clinical laboratories, including hospitals, independent labs, and research institutions, spanning different geographic regions as well as a survey of 50 Laboratory experts with a 50% response rate. Key metrics assessed included: -Frequency and duration of supply chain disruptions. -Financial costs incurred due to disruptions, including expenses related to expedited shipping, alternative sourcing, and inventory management. -Impact on laboratory workflow, turnaround times, and patient care outcomes. -Strategies employed to mitigate supply chain risks and enhance resilience. Results Our review revealed findings regarding the cost and importance of supply chain resilience in the clinical laboratory: -Supply chain disruptions were frequent, with 85% of laboratories experiencing at least one significant disruption in the past year and 100% in experienced disruptions 2022. -The average duration of disruptions was 2-3 weeks, leading to substantial delays in test processing and patient care. -Financial costs associated with disruptions averaged $50,000 per laboratory per annum, including expenses for rush orders, and premium-priced alternatives. -Common strategies employed were increased inventory or safety stocks, multi-vendor sourcing, and supply chain mapping. Conclusions The findings of this study underscore the critical importance of supply chain resilience in the clinical laboratory. Supply chain disruptions are not just inconveniences; they pose significant risks to patient safety, operational efficiency, and financial sustainability. Investing in supply chain resilience is imperative for laboratories to fulfill their mission of delivering timely and accurate patient results. To continue to enhance supply chain resilience, laboratories must adopt a proactive approach, leveraging data-driven insights and collaborative partnerships. This includes diversifying supplier networks and fostering closer collaboration between laboratory and procurement teams.

Inverted pendulum is a classical control problem.Traditional control methods such as PID and fuzzy control can achieve better results.With the rise of artificial intelligence technology, deep learning and reinforcement learning have attracted much attention.Among them, the combination of the two has considerable potential for in-depth reinforcement learning.Q-learning was proposed in 1989 as a classical algorithm in reinforcement learning. In recent years, some scholars proposed the algorithm of Deep learning plus q-learning, Deep Q network(DQN), to solve the problem that q-learning is inherently unable to solve the problem of high-dimensional input, and achieved good results.In this paper, based on VREP simulation environment, DQN is used to try to solve inverted pendulum problem.

Background: The importance of South African small businesses, especially retailers, in improving communities’ standards of living necessitates sound business strategies to survive and remain competitive. Selecting appropriate business strategies is cardinal for the sustainable development and growth of all types of businesses, including small businesses. The supply chain strategy defines how operations in the supply chain will support the overall competitive strategy and is orientated to promote either responsiveness or cost-efficiency. It is imperative that all inventory decisions be aligned with the orientation of the selected supply chain strategy.Objectives: The aim of this article was firstly to determine how small retailers operating within the boundaries of Soweto, a township in Gauteng, South Africa, manage their inventory decisions considering the orientation of the supply chain strategy; and secondly, to investigate whether retail industries differ in terms of inventory decisions.Method: A survey was conducted among 650 formal, independent small businesses operating within Soweto of which 556 were classified as retailers based on the Standard Industrial Classification. Structured interviewer-completed questionnaires were used to obtain quantitative data from owners or managers. Both descriptive and inferential analyses were performed to determine the orientation of the retailers’ supply chain strategy based on their inventory decisions relating to the average rate of purchasing inventory, the amount of inventory purchased, as well as the quality and variety of products offered to customers. Five hypotheses were formulated to determine whether a difference exists between six industry groups with regard to the management of inventory.Results: It was established that small township retailers predominantly manage their inventory decisions responsively. It was also concluded that differences exist in terms of how small township retailers manage their inventory based on the industry in which they operate. The results showed that retail hardware stores are focused on making inventory decisions more cost-efficiently than the other industry groups, whereas retail general stores are focused on making inventory decisions more responsively than the other industry groups.Conclusion: The owners or managers of small township retailers need to be educated in terms of inventory management decisions. This article can be used as an educational platform to show how small township retailers can identify the aligned inventory decisions for supporting a selected supply chain strategy orientation. Generally, small township retailers may maintain their competitive advantage if they align all business strategies to promote responsiveness. Since the industry in which small retailers operate influences inventory management decisions in terms of responsiveness and cost-efficiency, owners or managers must consider the specific industry in which they operate before making such decisions.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

Purpose This study aims to explore the interplay between supply chain collaboration and disruption and their joint impact on supply chain resilience. It also aims to identify the configurational conditions under which supply chain resilience is achieved, providing a holistic framework for enhancing supply chain management practices amidst disruptions. Design/methodology/approach This study uses a fuzzy-set qualitative comparative analysis (fsQCA) to investigate the complex causal mechanisms linking supply chain collaboration and disruption with supply chain resilience. The research integrates three dimensions of collaboration with three dimensions of disruption, offering a comprehensive understanding of supply chain resilience factors. Findings This study reveals that supply chain resilience is not solely dependent on individual elements but is shaped by unique configurations of collaboration and disruption factors. Information sharing emerges as a core factor in high resilience, while the absence of supply and facility disruptions significantly influences resilience outcomes. The research highlights the importance of multiple paths leading to supply chain resilience and the asymmetrical impact of collaboration and disruption. Research limitations/implications The study acknowledges limitations due to data sourced exclusively from China and the use of static data, suggesting the need for cross-temporal and international samples to enhance broader applicability. Future research should consider dynamic temporal changes and diverse theoretical perspectives to comprehensively examine the factors influencing supply chain resilience. Practical implications The research underscores the critical role of information sharing in bolstering supply chain resilience and advises firms to prioritize it in their strategies. It also highlights the importance of mitigating supply and facility disruptions through supplier diversification and robust contingency planning, offering actionable insights for enhancing operational efficiency and risk management in supply chain management. Social implications This research has significant social implications, particularly in the context of global supply chain disruptions. By identifying key factors that enhance supply chain resilience, it can help businesses better prepare for and respond to crises, thereby reducing economic instability and social disruption. Improved resilience can lead to more stable employment, maintain the flow of essential goods and services and contribute to overall societal well-being during times of supply chain stress. Originality/value This study offers original insights by applying fsQCA to explore the multifaceted relationship between supply chain collaboration, disruption and resilience. Its value lies in revealing the complex causal configurations that lead to high or low levels of supply chain resilience, challenging traditional linear perspectives and providing a nuanced understanding that can guide both academic research and practical supply chain management strategies.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.