Deep Reinforcement Learning Based Optimization Algorithm for Permutation Flow-Shop Scheduling

In this paper a discrete particle swarm optimization (DPSO) algorithm is proposed to solve permutation flowshop scheduling problems with the objective of minimizing the makespan. A discussion on implementation details of DPSO algorithm is presented. The proposed algorithm has been applied to a set of benchmark problems and performance of the algorithm is evaluated by comparing the obtained results with the results published in the literature. Further, it is found that the proposed improvement heuristic algorithm performs better when local search is performed. The results are presented.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

The permutation flow shop scheduling problem (PFSP), which is one of the most important scheduling types, is widespread in the modern industries. With the increase of scheduling scale, the difficulty and computation time of solving the problem will increase exponentially. Adding the knowledge to intelligent algorithms is a good way to solve the complex and difficult scheduling problems in reasonable time. To deal with the complex PFSPs, this paper proposes an improved simulated annealing (SA) algorithm based on residual network (SARes). First, this paper defines the neighborhood of the PFSP and divides its key blocks. Second, the Residual Network (ResNet) is used to extract and train the features of key blocks. And, the trained parameters are stored in the SA algorithm to improve its performance. Afterwards, some key operators, including the initial temperature setting and temperature attenuation function of SA algorithm, are also modified. After every new solution is generated, the parameters trained by the ResNet are used for fast ergodic search until the local optimal solution found in the current neighborhood. Finally, the most famous benchmarks including part of TA benchmark are selected to verify the performance of the proposed SARes algorithm, and the comparisons with the-state-of-art methods are also conducted. The experimental results show that the proposed method has achieved good results by comparing with other algorithms. This paper also conducts experiments on network structure design, algorithm parameter selection, CPU time and other problems, and verifies the advantages of SARes algorithm from the aspects of stability and efficiency.

The permutation flow-shop scheduling problem (PFSP) is an non-deterministic polynomialtime (NP) hard combinatorial optimization problems and has been widely researched within thescheduling community. In this paper, a memetic gravitation search algorithm (MGSA) is proposedto solve the PFSP for minimizing the makespan measure. The smallest position value (SPV) rule isutilized for converting the continuous number to job permutations for determining the most suitablethe proposed MGSA for the PFSP. The proposed MGSA uses a Nawaz-Enscore-Ham (NEH) heuristicalgorithm for initialization of population, and a simulated annealing (SA) is coupled with the variableneighborhood search (VNS) as the local search method to balance exploitation and exploration. Toverify the robustness of the MGSA, it is compared with three particle swarm optimization (PSO) algorithmson the basis of 12 PFSP instances with different job sizes ranging from 20 to 500. The resultsdemonstrate that the proposed MGSA can outperform other compared algorithms.

Designing an effective and efficient end-to-end optimization framework with good generalization for shop scheduling is an emerging topic in the informational manufacturing system. Existing end-to-end frameworks have achieved satisfactory results for COPs such as traveling salesman problem and vehicle routing problem. However, the performances of these methods in solving complex COPs such as shop scheduling need to be improved. In this paper, a knowledge-guided end-to-end optimization framework based on reinforcement learning is proposed to solve the permutation flow shop scheduling problem (PFSP). Firstly, a new policy network is designed based on the problem characteristics to deal with different scales of PFSPs and achieve iterative end-to-end generation. Secondly, an improved policy-based reinforcement learning algorithm by using the knowledge accumulated during the training process is designed to enhance the training quality. Thirdly, a knowledge-guided improvement strategy is introduced through the cooperation of local search and supervised learning to improve the learning of the policy. Simulation results and comparisons show that the knowledge-guided end-to-end optimization framework can obtain better results than different kinds of commonly-used optimization methods in limited computation time for solving the PFSP.

Scheduling problem has always been an important problem in the industrial sectors since it creates huge impact on the overall performance of the manufacturing. Good scheduling can reduce overall production time which then leads to lower cost and good resource management. The permutation flowshop scheduling (PFSP) is the classic scheduling problems that attracts both business and research area for almost half a century. On research side, a variety of complex algorithms have been introduced to solve the problems and provide high quality of solutions. Nevertheless, these algorithms will be useless if they fail to implement in practice where computational time and complexity of algorithm become an important issue of concern. This research proposes a Node-Based Coincidence Algorithm (NB-COIN) for the permutation flowshop scheduling problems (PFSP) aimed at Makespan minimization. NB-COIN is proved to be an effective algorithm that can provide good quality solutions using small amount of time and resources. The results generated by NB-COIN are also better than other well-known algorithms in consideration. Based on the bench-mark data sets of Taillard, 10% of the solutions provided by the presented algorithm are optimal solutions. Moreover, the solutions found by NB-COIN are also achieve 0.96% gap from upper bound in average. More importantly, those solutions are found within a short time.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

The marriage of operations research and reinforcement learning: Integration of NEH into Q-learning algorithm for the permutation flowshop scheduling problem

Aiming at Distributed Permutation Flow-shop Scheduling Problems (DPFSPs), this study took the minimization of the maximum completion time of the workpieces to be processed in all production tasks as the goal, and took the multi-agent Reinforcement Learning (RL) method as the main frame of the solution model, then, combining with the NASH equilibrium theory and the RL method, it proposed a NASH Q-Learning algorithm for Distributed Flow-shop Scheduling Problem (DFSP) based on Mean Field (MF). In the RL part, this study designed a two-layer online learning mode in which the sample collection and the training improvement proceed alternately, the outer layer collects samples, when the collected samples meet the requirement of batch size, it enters to the inner layer loop, which uses the Q-learning model-free batch processing mode to proceed, and adopts neural network to approximate the value function to adapt to large-scale problems. By comparing the Average Relative Percentage Deviation (ARPD) index of the benchmark test questions, the calculation results of the proposed algorithm outperformed other similar algorithms, which proved the feasibility and efficiency of the proposed algorithm.

Attempts to address the production scheduling problem thus far rely on simplifying assumptions, such as static environment and inflexible size of the problem, which compromises the schedule performance in practice due to many unpredictable disruptions to the system. Thus, the study of scheduling in the presence of real-time events, termed dynamic scheduling, continues to attract attention given the agility, flexibility, and timeliness modern production systems must deliver. Additionally, the changing nature of the manufacturing system also raises new challenges to existing scheduling strategies. At the front-end, the development of advanced data creation and exchange frameworks such as the Internet of things and cyber-physical system and their applications to the industrial environment have created an abundance of industrial data, while at the backend, edge and cloud computing technologies greatly enhance the capacity to process that data. Industrial data must be mined and analyzed so that the investment in infrastructure is not wasted, and the production system managed more effectively and in real-time. Many data-driven technologies have been adopted in scheduling research, a promising candidate among them being reinforcement learning (RL) which is able to build a direct mapping from observation of environment to actions that improve its performance. In this thesis, a deep multi-agent reinforcement learning (deep MARL) architecture is proposed to solve the dynamic scheduling problem (DSP). The deep reinforcement learning (DRL) algorithm is used to train the decentralized scheduling agents, to capture the relationship between information on the factory floor and scheduling objectives, with the aim of making real-time decisions for a manufacturing system with frequent unexpected events. Two major aspects of deep MARL application to DSP are addressed in this work, namely the conversion from traditional static scheduling problem (SSP) to dynamic scheduling in a practical context, and the adaptation of existing deep MARL algorithms to solve the scheduling problem in such an environment. Some impractical constraints of traditional studies are removed to create a research context that is closer to actual practice, result in a scheduling problem of variable size and scope. Specialized state and action representations that can handle the ever-changing specification of problem are developed; the criteria of feature selection in dynamic environment are also discussed. Recent progressions in DRL and MARL research are integrated into the proposed approach after selection and adaptation. In addition, various improvements to common deep MARL architecture are proposed, including the lightweight multilayer perceptron (MLP) encoder that is efficient in handling unstructured industrial data, a training scheme under the multi-agent architecture to improve the stability of training and overall performance, and knowledge-based reward-shaping techniques to decompose the joint reward signal into individual utilities to speed up the learning and encourage cooperative behavior between agents. Simulation studies are then conducted for the ablation study and validation. In the first stage, the performance of the proposed approach, either as individual components or as an integrated model, are tested in iterative simulation runs within which a unique instance of production is created. Meanwhile, a set of DRL-based approaches from recent publications are run in parallel. Results suggest that the contribution of each improvement is significant; the integrated architecture also delivers stronger performance than peer DRL-based approaches. For the validation, a set of priority rules that have strong performance in specified context and are widely applied in actual production scheduling are used as the benchmark. Proposed approach also provides performance gain compared to the strongest rule, with a minor increase in computation cost and negligible latency in decision-making.

The distributed permutation flow shop scheduling (DPFSS) is a permutation flow shop scheduling problem including the multi-factory environment. The processing times of the jobs in a real life scheduling problem cannot be precisely know because of the human factor. In this study, the process times and due dates of the jobs are considered triangular and trapezoidal fuzzy numbers for DPFSS environment. An artificial bee colony (ABC) algorithm is developed to solve the multi-objective distributed fuzzy permutation flow shop (DFPFS) problem. First, the proposed ABC algorithm is calibrated with the well-known DPFSS instances in the literature. Then, the DPFSS instances are fuzzified and solved with the algorithm. According to the results, the proposed ABC algorithm performs well to solve the DFPFS problems.

Permutation flow shop scheduling problem is a complicated global optimum problem. A number of intelligent algorithms have been used for this problem. In this paper, a plant growth simulation algorithm was used to solve permutation flow shop scheduling problem. The algorithm was experimented and the experimental results show that the Plant Growth Simulation Algorithm (PGSA)has better feasibility and validity for solving permutation flow shop scheduling problem.

The well-known particle swarm optimization (PSO) proposed by Kennedy and Eberhart has been widely applied to the continuous optimal problems. However, it is still intractable to apply PSO to discrete optimization problems, such as permutation flow shop scheduling problems (PFSSP). In this paper, a new high performing metaheuristic algorithm hybridizing PSO with variable neighborhood search (VNS) is proposed to solve PFSSP with the objective of minimizing makespan. NEH heuristic has been adopted in the first step to generate good solutions in the initial population, and then PSO and VNS are hybridized to search for optimal or near-optimal solutions of the PFSSP. Two effective neighborhood structures concerned with characteristics of PFSSP have been adopted to enhance VNS's performance. Computational experiments have been conducted on benchmarks and comparison results with other existing algorithms show the efficiency of the proposed algorithm.

In the domain of continuous control, deep reinforcement learning (DRL) demonstrates promising results. However, the dependence of DRL on deep neural networks (DNNs) results in the demand for extensive data and increased computational cost. To address this issue, a novel hybrid actor-critic reinforcement learning (RL) framework is introduced. The proposed framework integrates the broad learning system (BLS) with DNN, aiming to merge the strengths of both distinct architectural paradigms. Specifically, the critic network employs BLS for rapid value estimation via ridge regression, while the actor network retains the DNN structure to optimize policy gradients. This hybrid design is generalizable and can enhance existing actor-critic algorithms. To demonstrate its versatility, the proposed framework is integrated into three widely used actor-critic algorithms-deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and twin delayed DDPG (TD3), resulting in BLS-augmented variants. The experimental results reveal that all BLS-enhanced versions surpass their original counterparts in terms of training efficiency and accuracy. These improvements highlight the suitability of the proposed framework for real-time control scenarios, where computational efficiency and rapid adaptation are critical.

ABSTRACTA Cuckoo Search (CS) algorithm based on ant colony algorithm is proposed for scheduling problem in permutation flow shop scheduling problem (PFSP). When the raised CS algorithm obtains the position of the bird nest to be updated, it is used as a set of initial solution of the ant colony optimization algorithm (ACO), and ACO algorithm search optimization is performed in a very small range. After that, the solution obtained by the ACO search is taken as a new candidate solution, compared with the candidate bird nest according to the fitness degree. When the candidate solution of the ACO search optimization is better than the one generated by the Lévy flight, the latter is replaced. Finally, the CS algorithm is selected, changing the new bird nest position according to the abandonment probability. The updated position tends to be more optimal, which improves the quality of the solution as well as the convergence speed and accuracy of the algorithm. Comparing the performance of the proposed algorithm with the standard Cuckoo one, by testing function, the optimized performance was verified. Finally, the Car benchmark test served as test data, and the performance in the PFSP was compared. The effectiveness and superiority in the algorithm in solving problem were confirmed.