Reinforcement learning applied to production planning and control

PurposeThe aim of this paper is to identify some specificities of production planning and control (PPC) activities in the one-of-a-kind-production (OKP) process through an extensive literature review. Relevant aspects related to systems and PPC activities in the context of OKP environment are discussed, and six opportunities for future research are highlighted.Design/methodology/approachThe following research is based on a review of 53 articles published in peer-reviewed journals over the past three decades. After an initial descriptive analysis based on bibliometric indicators, a cluster analysis of 15 most cited articles was carried out using multivariate data analysis techniques and in-depth analysis.FindingsThe results reveal some specificities inherent to the clusters featured in the research, including aspects of planning, control and systems for OKP process. This cluster addresses information regarding next-generation manufacturing systems, scheduling and design science, computer simulation and project approach. On the other hand, the authors point out six topics for future research regarding contemporary issues associated with PPC in the context of OKP.Originality/valueThis paper fills an important gap regarding OKP production planning and control practices. The results provide a theoretical overview of different PPC practices suitable for the OKP environment. Furthermore, it can provide insights for scientific developments in order to manage the complexity inherent in the OKP process.

The rapid advancement of autonomous robots in domains such as healthcare, transportation, and disaster response necessitates robust ethical decision-making frameworks to mitigate risks associated with uncertain environments. This research delves deeply into the application of reinforcement learning (RL) for enabling "safe" decisions in robots facing multiple risky outcomes. By synthesizing over 80 peer- reviewed sources, including recent 2024-2025 surveys on ethical RL and safe RL benchmarks, it systematically addresses three pivotal research questions: (1) How should a robot choose between two risky paths? (2) Can RL augmented with reward shaping effectively reduce harm in unpredictable environments? (3) How do prominent RL algorithms compare in terms of safety efficacy in robotic applications? Key findings indicate that multi-objective RL (MORL) and constrained policy optimization variants, such as Proximal Policy Optimization (PPO) with Lagrangian constraints, outperform traditional RL in balancing ethical imperatives with task performance, achieving up to 40% reductions in violation rates in simulated benchmarks. Reward shaping, particularly potential-based and ethics-informed variants, accelerates ethical convergence by 3-6x while curbing reward hacking—a common pitfall where agents exploit reward loopholes. A novel student-led simulation in a custom gridworld environment demonstrates that shaped rewards enhance average returns by approximately 12% with near-zero harm incidents over 1,000 episodes. This expanded study proposes an integrated Ethical Safe RL (ESRL) framework, incorporating human-in-the-loop feedback and verifiable constraints, tailored for accessible implementation in educational settings. It underscores persistent challenges like distributional shifts in real-world deployment and calls for interdisciplinary ethics integration. By bridging theoretical surveys with empirical simulations, this work empowers high-school researchers to contribute to responsible AI robotics.

This study aims to provide a profound picture of where existing production planning and control (PPC) concepts fail in small to medium sized make-to-order (MTO) companies. While several studies have discussed the specific requirements of small and medium sized enterprises (SMEs) through surveys or observations in order to determine the suitability of existing PPC concepts, in general, in-depth empirical evidence is rare. In this article, seven case study companies are studied with the backbone of the analysis based on order progress data collected from each company. In each case, logistic performance problems are determined and traced back to their root causes. To distinguish between common problems and company specific elements, a comparative analysis across the cases has been performed. The analysis shows that a significant proportion of common PPC problems can be attributed to a limited set of decision points in the flow of orders. Furthermore, we find that most performance losses can actually be predicted before an order is released to the shop floor. Decisions after release are mainly made in order to correct for the delays caused before release. In the pre-production phase, before release, two common problem areas are identified: (1) inadequate capacity planning overviews to support sales decisions, and (2) uncontrolled delays in engineering. Observed problems that appear after release tend to be more diverse and company-specific. A review of available PPC concepts for MTO SMEs indicates that these areas have been neglected in the literature to date, leading to our call for a new focus in PPC research.

At present, it is still impossible to accurately solve realistic problems of production planning and control (PPC) in an acceptable period of time. Therefore numerous heuristic procedures were developed to solve problems of PPC. The efficiency of these procedures is usually appraised by means of statistical analyses. The data necessary for the analyses must meet high requirements to obtain significant and generalizable results from the statistical analyses. A procedure for the generation of test data is presented in the article. The procedure is based on given characteristics. It stochastically produces PPC data of arbitrary complexity. The procedure for the generation of test data is based on a hierarchical concept. The user enters values of aggregated classification numbers which globally characterize the modeled production system and the amount of released production orders. The values of detailed classification numbers are stochastically generated from the aggregated classification numbers. These values form the input for the generation of the PPC data wanted. This generation is stochastically, too. The user may modify both classification numbers and PPC data in order to generate test data equivalent to known practical data or to acquire test data with slightly changing characteristics for analyses of parameter sensitivity.

Production Planning and Control for Sustainable Management Systems

Inventory management remains a cornerstone of effective supply chain performance, directly influencing cost efficiency, service quality, and organizational agility. In today’s hypercompetitive and uncertain market environment, inventory decisions must account for complex variables such as fluctuating demand, supply disruptions, lead time variability, and market seasonality. Traditional inventory control models such as the Economic Order Quantity (EOQ), base-stock policies, and (s, S) strategies are often static in nature. They rely on pre-defined parameters and assume stationarity in demand and supply, limiting their ability to respond dynamically to real-time changes. In contrast, Reinforcement Learning (RL) offers a paradigm shift in how inventory decisions can be optimized. As a subfield of machine learning, RL enables agents to learn optimal strategies through repeated interactions with an environment, using trial-and-error exploration and reward-based feedback. RL agents can observe the system state (e.g., inventory levels, demand signals, lead time status), choose actions (e.g., place an order or wait), and receive feedback in the form of rewards (e.g., service level achievements or cost penalties), thus iteratively improving their policies. This study explores how RL can be applied to optimize inventory management in environments characterized by uncertainty and real-time decision-making needs. Specifically, we investigate how different RL algorithms such as Q-learning, Deep Q Networks (DQN), and Policy Gradient methods perform in various inventory scenarios. Additionally, we examine the computational and operational implications of deploying RL in real-world settings, including issues of model convergence, exploration-exploitation tradeoffs, data requirements, and scalability. We also discuss how RL can complement other AI techniques such as demand forecasting models and predictive analytics in creating end-to-end intelligent supply chain solutions. By bridging the gap between theoretical RL frameworks and practical inventory management applications, this paper contributes to both the academic literature and industrial practice. Our goal is to demonstrate that RL is not only a theoretically elegant solution but also a viable tool for achieving inventory efficiency and supply chain resilience.

Reinforcement Learning (RL) is a successful and established Artificial Intelligence (AI) method, particularly with recent groundbreaking progress in Deep Reinforcement Learning (DRL). Reinforcement learning is very well suited for sequential decision-making tasks, wherein a learned agent learns an optimal policy after many interactions with an environment. The present paper examines the application of reinforcement learning for automating screening of literature in academic research, particularly in the fields of computer science and e-learning. Keyword filtering techniques, while predominantly applied, are found to be inflexible as well as unable to capture the dynamic nature of research themes. To overcome such constraints, we present a Deep Q-Network (DQN)-based reinforcement learning model that combines reinforcement learning with the Semantic Scholar API to enhance research paper classification based on dynamically acquired decision rules. The proposed reinforcement learning model was trained and tested with a dataset of 8,934 research papers, accessed by systematic searching. The agent filtered and picked 11 effective papers depending on improved selection criteria like publication date, keyword relevance, and scholarly topic provided. The model iteratively optimizes the decision-making process through reward-based learning and therefore maximizes categorization accuracy over time. Test experiments demonstrate utilization of RL-based suggested framework yields classification accuracy at 91.5%, recall at 86.3%, and precision at 89.7%. A comparison test demonstrates that the approach performs 12.5% better on recall and 9.8% better on accuracy compared to traditional keyword-filtering methods. The finding confirms the power of the model in minimizing false positives and false negatives for screening literature, hence proving the scalability and adaptability of reinforcement learning in managing high academic data. This work offers a scalable, cognitive approach to conducting systematic reviews of literature through the application of reinforcement learning to programmatically execute work in academic research. The work shows the promise of reinforcement learning to further enhance research methodology, make literature reviews more effective, and facilitate more knowledgeable decision-making in fast-changing scientific disciplines. Further research will be focused on incorporating hybrid AI models with multi-agent systems of reinforcement learning for responsiveness and classification enhancements.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Introduces concepts for the integration of Computer‐Aided Process Planning (CAPP) and Production Planning and Control (PPC). Characterizes the current situation regarding integration by a one‐way communication from CAPP to PPC, since a revised link is not envisaged. Introduces a new approach to full integration by the extension of functions within both systems′ components as well as integration of both data and information technology.

The objective of this research is to analyze the Production Planning and Control (PPC) as a dissemination tool of the operation strategy in industries that compete in environmental sustainability. We used a qualitative methodology supported by cases studies, based on interviews with managers responsible for the production planning and control of four big Brazilian industries. The results indicated that PPC can be an efficient tool to dissiminate the operation strategy for its power of integration with the many key areas involved in an industry, what can bring pratical implications to the ones responsible for the formulation and implementation of the operation strategy. The result is of value because PPC can be used as an ally of the operation/production managers to promote and lead the implementation of the operation strategy on the shop floor.

Production Planning and Control (PPC) requires human decision making in several process steps like production programme planning, production data management and performance measurement. Thereby, human decisions are often biased leading to an aggravation of logistic performance. Exemplary, the lead time syndrome (LTS) shows this connection. While production planners aim to improve due date reliability by updating planned lead times the result is even a decreasing due date reliability. In current research in the field of production logistics the impact of cognitive biases on the decision-making process in PPC remains at a silent place. We aim to close the research gap by combining a systematic literature review on behavioral operations management as well as cognitive biases and applying the Aachen PPC model. Based on a case study from the steel industry we show the influence of cognitive biases on human decision making in several phases of PPC.

Germany pushes its energy industry towards renewable energy sources which results in an increase of distributed energy generation. Therefore, energy supply will fluctuate more and more and companies need to react to higher and volatile prices as well as to a fluctuating supply of energy. One reaction is to reactively consider these changing prices and supply of energy within production planning and control (PPC). PPC therefore acts as a control unit to integrate volatile energy supply into the production process. Within this paper a scenario is considered where a virtual power plant (VPP) combines different types of distributed energy generation. Energy simulation is then used to demonstrate how fluctuating energy prices can be considered within PPC.

While the recent advanced military operational concept requires an intelligent support of command and control, Reinforcement Learning (RL) has not been actively studied in the military domain. This study points out the limitations of RL for military applications from literature review and aims at improving the understanding of RL for military decision support under the limitations. Most of all, the black box characteristic of Deep RL makes the internal process difficult to understand in addition to complex simulation tools. A scalable weapon selection RL framework is built which can be solved either by a tabular form or a neural network form. The transition of the Deep Q-Network (DQN) solution to the tabular form makes it easier to compare the result to the Q-learning solution. Furthermore, rather than using one or two RL models selectively as before, RL models are divided as an actor and a critic, and systematically compared. A random agent, Q-learning and DQN agents as a critic, a Policy Gradient (PG) agent as an actor, Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) agents as an actor-critic approach are designed, trained, and tested. The performance results show that the trained DQN and PPO agents are the best decision supporter candidates for the weapon selection RL framework.

Definition: Efficient and scalable exploration of complex problem spaces.Methods Used: Reinforcement learning (RL) and surrogate models.Problems: 1) Accelerator control for Booster at Fermi National Laboratory (FNAL), 2) Block copolymer (BCP) self-annealing control, 3) Electrical Power Grid (current work) Software: EXARL scalable RL framework and applications: 1) Neural-network-based digital twin of FNAL Booster, 2) BCP partial differential equation (PDE)-based simulations Results: Open Source EXARL scalable framework.Functioning RL applications using EXARL: 1) ExaBooster; 2) ExaCH (BCP control).EXARL testing via Bsuite and PyTest.Next Steps: Plans for collaboration on RL for fusion energy facility control.EXARL scalable RL system components and workflow.

This work describes a model for distributed dynamic Production Planning and Control (PPC) agent based system, which includes interoperation with manufacturing automation. It is presented a demonstration prototype involving distributed software agents and industrial equipment integration, which implements part of the developed model functionalities. Clients can send orders, and resources may apply for those orders fulfilment. Resources with orders allocated to, start automatically the required manufacturing operations. The prototype was implemented integrating several tools, including LabVIEW and LEGO Mindstorms components. This is useful to validate the integration, proposed by the dynamic PPC model, between production planning processes and manufacturing execution operations.