Solving large scale disassembly line balancing problem with uncertainty using reinforcement learning

Disassembly lines should be balanced efficiently to increase productivity of the line and to reduce disassembly costs. This problem is called disassembly line balancing problem (DLBP). The objective of the DLBP is usually to find the minimum number of disassembly workstations required. This study introduces parallel DLBP (PDLBP) with single-product and proposes a network model based on the shortest route model (SRM) for solving PDLBP. The proposed model is illustrated via numerical examples. A comprehensive experiment is also conducted to evaluate problem-specific features of disassembly lines. To the best of our knowledge, this is the first study dealing with PDLBP. This paper will present a different point of view regarding DLBP.

With green and smart manufacturing emergence and development, effective conduct with End of Life (EOL) products is an extremely important and hot research topic. Disassembly line design is a fundamental and complex problem in EOL product recycling and remanufacturing, especially when considering uncertainty. In this paper, we investigate a disassembly line balancing problem (DLBP) with stochastic task processing time to minimize the cycle time. We focus on selecting alternative task processes, determining the number of open workstations and assigning operational tasks to the workstations. A joint chance-constrained programming is proposed and we adopt a distribution-free method, which is based on Markov Inequality, to approximate the chance constraints as the distribution is difficult to fully estimated. Risk-neutral model may provide solutions that perform poorly while considering uncertainty. Therefore, a risk-averse measure called Conditional Value-at-Risk (CVaR), which is rarely used in the DLBP, is considered in this paper to incorporate decision makers’ perspectives. To the best of our knowledge, we are the first to consider risk-averse model in DLBP with minimizing cycle time under uncertain task processing time. Finally, computational experiments are conducted and give the impacts by testing different risk levels and confidence levels.

Multi-objective partial parallel disassembly line balancing problem using hybrid group neighbourhood search algorithm

Recently, several mathematical programming formulations and solution approaches have been developed for the stochastic disassembly line balancing problem (DLBP). This paper aims at finding optimal solutions for the stochastic DLBP. Two second-order cone programming (SOCP1 and SOCP2) models and five piecewise linear mixed integer programming (PwLP) models are presented. The PwLP formulations involve two specially ordered sets of type 2 (S1 and S2) models and three convex combination (CC1, CC2 and CC3) models. In each modelling category, the latter models strengthen the initial S1 and CC1 models. Our computational analysis of a total 240 instances of ten problems demonstrates that all the seven models can be used to solve practical-sized DLBP problems to optimality using GUROBI. The SOCP2 model and the strengthened S2 and CC2 models lead to lower computation times, compared to SOCP1, S1, CC1 and CC3, respectively. Using the strengthened S2 and CC2 formulations, the CPU times of the CC3 model available in the literature can be reduced by 50 and 40%, respectively. Besides analysing the optimal solutions and the differences of the computation times, we present insights gained from our results.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Active screening is a common approach in controlling the spread of recurring infectious diseases such as tuberculosis and influenza. In this approach, health workers periodically select a subset of population for screening. However, given the limited number of health workers, only a small subset of the population can be visited in any given time period. Given the recurrent nature of the disease and rapid spreading, the goal is to minimize the number of infections over a long time horizon. Active screening can be formalized as a sequential combinatorial optimization over the network of people and their connections. The main computational challenges in this formalization arise from i) the combinatorial nature of the problem, ii) the need of sequential planning and iii) the uncertainties in the infectiousness states of the population. Previous works on active screening fail to scale to large time horizon while fully considering the future effect of current interventions. In this paper, we propose a novel reinforcement learning (RL) approach based on Deep Q-Networks (DQN), with several innovative adaptations that are designed to address the above challenges. First, we use graph convolutional networks (GCNs) to represent the Q-function that exploit the node correlations of the underlying contact network. Second, to avoid solving a combinatorial optimization problem in each time period, we decompose the node set selection as a sub-sequence of decisions, and further design a two-level RL framework that solves the problem in a hierarchical way. Finally, to speed-up the slow convergence of RL which arises from reward sparseness, we incorporate ideas from curriculum learning into our hierarchical RL approach. We evaluate our RL algorithm on several real-world networks. Results show that our RL algorithm can scale up to 10 times the problem size of state-of-the-art (the variant that considers the effect of future interventions but unscalable) in terms of planning time horizon. Meanwhile, it outperforms state-of-the-art (the variant that scales up but does not consider the effect of future interventions) by up to 33% in solution quality.

Disassembly operations are inevitable elements of product recovery with the disassembly line as the best choice to carry out the same. In the light of different structures of returned products (models) and variations in task completion times, the process of disassembly could not be efficiently mapped on a simple straight line. Another important issue that needs consideration is the task-time variability pertaining to human factor. In order to resolve these complexities a Mixed-Model U-shaped Disassembly Line with Stochastic Task Times has been proposed in this article. A novel approach, Collaborative Ant Colony Optimization (CACO), has been utilized that simultaneously tackles the interrelated problem of line balancing and model sequencing. The distinguishing feature of the proposed approach is that it maintains bilateral colonies of ants which independently identifies the two sequences, but utilizes the information obtained by their collaboration to guide the future path. The approach is tested on benchmark instances that were generated using Design of Experiment techniques and Analysis of Variance is performed to determine the impact of various factors on the objective. The robustness of proposed algorithm is authenticated against Ant Colony Optimization over which it always demonstrated better results thereby proving its superiority on the concerned problem.

To find the path that minimizes the time to navigate between two given points in a fluid flow is known as Zermelo's problem. Here, we investigate it by using a Reinforcement Learning (RL) approach for the case of a vessel that has a slip velocity with fixed intensity, Vs, but variable direction and navigating in a 2D turbulent sea. We show that an Actor-Critic RL algorithm is able to find quasioptimal solutions for both time-independent and chaotically evolving flow configurations. For the frozen case, we also compared the results with strategies obtained analytically from continuous Optimal Navigation (ON) protocols. We show that for our application, ON solutions are unstable for the typical duration of the navigation process and are, therefore, not useful in practice. On the other hand, RL solutions are much more robust with respect to small changes in the initial conditions and to external noise, even when Vs is much smaller than the maximum flow velocity. Furthermore, we show how the RL approach is able to take advantage of the flow properties in order to reach the target, especially when the steering speed is small.

Reinforcement learning for a class of continuous-time input constrained optimal control problems

This paper introduces the integrated disassembly line balancing and routing problem (I-DLB-RP). The I-DLB-RP simultaneously optimises two well-known problems. The former one balances the disassembly lines in the disassembly centres, whereas the latter one constructs a routing plan to distribute the usable components, generated by the disassembly process, from disassembly centre to the remanufacturing centres, i.e. customers. With the increasing importance of the disassembly process for tackling with the burden of waste and the number of disassembled products, the distribution planning of usable components released after the disassembly process becomes essential. This paper considers several scenarios: single-component distribution, multi-component distribution, inventory cost, and multi-period conditions. We propose five linear and non-linear mathematical models. Extensive computational experiments conducted on generated realistic benchmark instances. The analyses quantify the benefits of integrating the two problems.

In the ever-changing world of decision-making, when game theory and reinforcement learning(RL) come together, they create a fascinating combination that shows a new way to solve complex problems in many different fields. The combination of game theory and RL is a powerful convergence that opens up a hopeful new frontier for dealing with difficult decision-making problems in many different fields. Research on the convergence of game theory and RL has shown to be beneficial, providing essential insights into challenging decision-making issues in various disciplines. This study investigates the recent developments of game theory and RL approaches through a systematic review and highlights the significance of game theory in boosting reinforcement algorithms and increasing the interaction of autonomous vehicles, safeguarding edge caching, and more. It offers a thorough account of the developments at the confluence of game theory and RL. The reviewed papers mainly focus on broad themes and address three important research questions: the impact of game theory on multi-agent reinforcement learning (MARL), the significant contributions of game theory to RL, and the significant impact areas. Following the methodology, search outcomes, and study areas is a discussion on game theory-related terminology, followed by study findings. The review’s conclusions are offered with ideas for further study and open research questions. The importance of game theory in advancing MARL, the potential of game theory in promoting RL strategies, and the opportunities for combining game theory and RL in cutting-edge fields like mobile edge caching and cyber-physical systems(CPS) are all emphasized in the conclusion. This review article advances our knowledge of the theoretical underpinnings and real-world applications of game theory and RL, laying the groundwork for future improvements in decision-making techniques and algorithms.

Experimental research such as cell cultures and carbon nanotube (CNT) growth are largely governed by following predefined execution protocols with fine-tuned control of parameters. There are promising opportunities to apply reinforcement learning (RL), an established learning technique in the area of artificial intelligence, in order to automate CNT growth process and accelerate related scientific breakthroughs in material discovery. Although there are benefits in RL-based exploration and exploitation methodologies, there are also challenges in developing relevant learning policies in experimental settings relating to CNT growth. In this paper, we present a novel data-driven RL approach for assisting experimental CNT growth. Our approach focuses on developing an RL model to learn from simulation-based images and characteristics of temporal CNT growth, considering various growth parameters. Our RL model learns from CNT growth variation in a simulation-based environment where critical control parameters, such as density, growth rate, tube radius, tube stiffness, and Van der Waals forces, are used. Our RL model enables CNT growth automation in order to explore of a wider range of growth conditions, and improve reproducibility. The ultimate goal of our RL model is to achieve desired CNT growth by dynamically controlling growth parameters throughout a sequence of experiments. We evaluate the effectiveness of our RL approach by measuring the improvement in the maximum compressive strength of a carbon nanotube ‘with’ and ‘without’ the RL model. Our results show the effectiveness of course correction recommended by our RL approach when controlling the parameters of angular deviation and rate of growth of carbon nanotubes, when compared against non-regulated CNT growth.

With the advancement of science and technology in recent years, the rate of product upgrading by end users has increased, resulting in a high number of End-Of-Life (EOL) products. For environmental protection and economic benefits, disassembly lines are set up to recycle them. A disassembly line balancing problem arises and has attracted widespread attention. In this paper, we design a model to address a class of disassembly line balancing problems in which the main optimization objective is to maximize profits, while keeping carbon emission in a tolerable range. A discrete whale optimization algorithm is proposed to search their high-quality solutions. Its efficiency is verified by comparing its solutions with their analytic solutions obtained from a commercial optimization solver and other swarm intelligence algorithms. Experimental results show that it is significantly faster than CPLEX and other swarm intelligence algorithms in solving large-scale optimization problems while guaranteeing its solutions quality.

Reinforcement learning methods have recently achieved impressive results on a wide range of control problems. However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution. This limits their applicability to complex input spaces such as video signals, and makes them impractical for use in complex real world problems, including many of those for video based control. Supervised learning, on the contrary, is capable of learning on a relatively limited number of samples, but relies on arbitrary hand-labelling of data rather than task-derived reward functions, and hence do not yield independent control policies. In this article we propose a novel, model-free approach, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments. We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality. Additionally, we demonstrate applicability of the method to MuJoCo control problems.

Mobile Edge Computing (MEC) is a promising paradigm to support high-quality time-sensitive applications. In this paper, we investigate the service migration (i.e., whether, when, and where to migrate the services) to seamlessly serve mobile users in small-cell MEC systems. The service migration is formulated as an optimization problem to minimize the long-term system average delay that consists of queuing, communication, and migration delays. Considering the dynamic user mobility and network conditions, the formulated problem is non-convex and difficult to solve in real time. To this end, we propose a Mobility-aware Service Migration scheme, named MSM, to make real-time decisions on service migrations by utilizing reinforcement learning (RL) approaches. Specifically, we first design a user classification mechanism based on users’ mobility patterns to reduce the complexity of decision-making. We then formulate the service migration as a Markov decision process and devise an RL-based framework to make service migration decisions in real time in the dynamic MEC environment. Extensive data-driven experiments demonstrate the efficacy of MSM in reducing the system average delay.