Application of Markov decision processes to search problems

Tabu search is one of the most effective heuristics for locating high-quality solutions to a diverse array of NP-hard combinatorial optimization problems. Despite the widespread success of tabu search, researchers have a poor understanding of many key theoretical aspects of this algorithm, including models of the high-level run-time dynamics and identification of those search space features that influence problem difficulty. We consider these questions in the context of the job-shop scheduling problem (JSP), a domain where tabu search algorithms have been shown to be remarkably effective. Previously, we demonstrated that the mean distance between random local optima and the nearest optimal solution is highly correlated with problem difficulty for a well-known tabu search algorithm for the JSP introduced by Taillard. In this paper, we discuss various shortcomings of this measure and develop a new model of problem difficulty that corrects these deficiencies. We show that Taillard's algorithm can be modeled with high fidelity as a simple variant of a straightforward random walk. The random walk model accounts for nearly all of the variability in the cost required to locate both optimal and sub-optimal solutions to random JSPs, and provides an explanation for differences in the difficulty of random versus structured JSPs. Finally, we discuss and empirically substantiate two novel predictions regarding tabu search algorithm behavior. First, the method for constructing the initial solution is highly unlikely to impact the performance of tabu search. Second, tabu tenure should be selected to be as small as possible while simultaneously avoiding search stagnation; values larger than necessary lead to significant degradations in performance.

Solving job shop scheduling problems via deep reinforcement learning

Content caching provides a rare trade-off opportunity between low-cost memory and energy consumption, yet finding the optimal causal policy with low computational complexity remains a challenge. In this paper, we formulate the problem of joint pushing and caching at end users in a Markov decision process (MDP) framework with the aim of minimizing energy cost under causal knowledge of the user request delay information and strict delay constraint. We put forth a novel approach to decouple the influence of buffer occupancy and user request, allowing one to equivalently run value iteration in a much smaller state space. Furthermore, by exploiting a structural property we discovered as generalized monotonicity , we design a computationally efficient non-iterative algorithm, fast assignment of state transition, to map the optimal policy from the smaller state space back to the complete state space. This, in turn, significantly simplifies the search for optimal policy. The results attain close performance in comparison with theoretical bounds from non-causal policies, while benefiting from better time efficiency than the unadapted MDP solutions.

Multicasting is an efficient technique to simultane-ously transmit common messages from the base station (BS) to multiple mobile users (MUs). The multicast scheduling problem for multiple messages over multiple channels, which jointly minimizes the energy consumption of the BS and the latency of serving asynchronized requests from the MUs, is formulated as an infinite-horizon Markov decision process (MDP) with large discrete action space and multiple time-varying constraints. By studying the intrinsic features of this MDP under stationary policies and refining the reward function, we first simplify it to an equivalent form with a much smaller state space. Then, we propose a modified deep reinforcement learning (DRL) algorithm, namely the permutation-based Wolpertinger deep deterministic policy gradient (PW-DDPG), to solve the simplified problem, which utilizes a permutation-based action embedding module to address the large discrete action space issue and a feasible exploration module to deal with the time-varying constraints. Moreover, as a benchmark, an upper bound of the considered MDP is derived by solving an integer programming problem.

Solving combinatorial optimization problems with single seekers society algorithm

Discrete Artificial Algae Algorithm for solving Job-Shop Scheduling Problems

Markov decision process (MDP) models are widely used for modeling sequential decision-making problems that arise in engineering, economics, computer science, and the social sciences. Many real-world problems modeled by MDPs have huge state and/or action spaces, giving an opening to the curse of dimensionality and so making practical solution of the resulting models intractable. In other cases, the system of interest is too complex to allow explicit specification of some of the MDP model parameters, but simulation samples are readily available (e.g., for random transitions and costs). For these settings, various sampling and population-based algorithms have been developed to overcome the difficulties of computing an optimal solution in terms of a policy and/or value function. Specific approaches include adaptive sampling, evolutionary policy iteration, evolutionary random policy search, and model reference adaptive search. This substantially enlarged new edition reflects the latest developments in novel algorithms and their underpinning theories, and presents an updated account of the topics that have emerged since the publication of the first edition. Includes: innovative material on MDPs, both in constrained settings and with uncertain transition properties; game-theoretic method for solving MDPs; theories for developing roll-out based algorithms; and details of approximation stochastic annealing, a population-based on-line simulation-based algorithm. The self-contained approach of this book will appeal not only to researchers in MDPs, stochastic modeling, and control, and simulation but will be a valuable source of tuition and reference for students of control and operations research.

Solving non-convex economic load dispatch problem via artificial cooperative search algorithm

Many problems today need the computing power that is only available by using large‐scaleparallel processing. For a significant number of these problems, the density of theglobal communications between the individual processors dominates the performance ofthe whole parallel implementation on a distributed memory multiprocessor system. In thesecases, the design of the interconnection network for the processors is known to play asignificant part in the efficient implementation of real problems. Important criteria to optimisethe efficiency of a configuration are the maximum and average distance a message hasto travel between processors. Minimum path systems are irregular multiprocessor computerarchitectures which optimise these criteria. These architectures provide an efficient alternativeto the more common regular topologies for solving real applications in parallel. Thispaper presents new results for two combinatorial problems that occur during the generationof these optimal irregular configurations. These are: (1) The design of the optimum interconnection network between the processors for configurationscontaining up to 128 processors. (2) The design of the routing tables to provide the optimal routing of messages withinthese irregular networks. The paper shows how these combinatorial problems have been solved, using genetic algorithmsfor the first problem and a random local search procedure for the second. It alsoincludes a comparison with the results obtained for regular topologies, for example: hypercubes,tori, and rings.

Jobshop scheduling is a classic instance in the field of production scheduling. Solving and optimizing the scheduling problem of the jobshop can greatly reduce the production cost of the workshop and improve the processing efficiency, thereby improving the market competitiveness of the manufacturing enterprises. In order to make decisions on the complex dynamic scheduling process more accurately and simplify the solution process, the jobshop scheduling problem can be transformed into a reinforcement learning problem based on the Markov decision process. The performance of the adaptive scheduling algorithm in a dynamic manufacturing environment is improved based on the Deep Q Network (DQN). In the proposed scheduling algorithm, five state features of continuous value ranges are designed for input to a Deep Neural Network (DNN), as well as ten well-known heuristic dispatching rules are selected as the action set of the DQN. In the proposed scheduling algorithm, the target network and the prediction network are used to train the parameters. An action selection strategy based on the “softmax” function is designed in DQN. It selects dispatching rules with the largest action value as the execution action, thereby solving the problem that the suboptimal action value is greater than the optimal action Q value in the early learning stage. Furthermore, the non-optimal action is selected with a greater probability in the later learning stage. Ten benchmark jobshop test instances called “LA” used as simulation objects and operated in a simulation environment composed of Python. The simulation results confirm that the proposed scheduling algorithm based on DQN has better performance and universality than a single dispatching rule or traditional Q learning algorithm.

Contributors

Ant-TD: Ant colony optimization plus temporal difference reinforcement learning for multi-label feature selection

An optimization algorithm based on the ‘divide-and-conquer’ methodology is proposed for solving large job shop scheduling problems with the objective of minimizing total weighted tardiness. The algorithm adopts a non-iterative framework. It first searches for a promising decomposition policy for the operation set by using a simulated annealing procedure in which the solutions are evaluated with reference to the upper bound and the lower bound of the final objective value. Subproblems are then constructed according to the output decomposition policy and each subproblem is related to a subset of operations from the original operation set. Subsequently, all these subproblems are sequentially solved by a particle swarm optimization algorithm, which leads directly to a feasible solution to the original large-scale scheduling problem. Numerical computational experiments are carried out for both randomly generated test problems and the real-world production data from a large speed-reducer factory in China. Results show that the proposed algorithm can achieve satisfactory solution quality within reasonable computational time for large-scale job shop scheduling problems.

In our previous research applied to the job shop scheduling problem (JSSP) for machine tool production, the multi-objective optimisation model based on the particle swarm optimization (PSO) research had an imbalance performance between the convergence rate and the convergence precision. In this article, a new hybrid algorithm using PSO and generic algorithm (GA) is proposed to solve this particular problem. In this new algorithm, named hybrid PSO-GA algorithm (HPGA), the PSO algorithm is redefined and modified by introducing genetic operators, i.e. the crossover operator and the mutation operator, to update the particles in the population. The HPGA is then applied in heavy machinery company with minimising machines’ makespan and minimising jobs’ tardiness as the two optimal objectives. The comparisons with actual application report have illustrated that the proposed HPGA can obtain higher quality of schedule solution for machine tool production. Furthermore, with solution quality and convergence rate as the two estimation measurements metrics, some comparisons are performed in order to illustrate that the HPGA has superiority over the PSO, GA and simulated annealing algorithm (SA). Results have indicated that, with the combination of the merits of PSO and GA, the proposed HPGA approach can achieve not only better solution quality but also faster convergence rate than the PSO, GA and SA, within a reasonable computation time for high dimensions JSSP.

Harmony Search algorithm (HSA) is a general-purpose heuristic algorithm that has been widely applied to resolve many combinatorial optimization problems. Since it was developed, many researches focused on improving the algorithm search capabilities through proposing a number of HSA variants. This work aims to focus on improving the investigation process of HSA that is assigned to random consideration part of the algorithm (i.e. random search operator). A new technique is proposed to enhance the exploration behavior of the algorithm, where the search range (zone) of optimization problem is divided randomly into n number of sub-zones. The algorithm will then run through all defined sub-zones to generate n initial harmonies for each sub-zone. To determine the best initial harmonies along with their belonging sub-zones, the average of the fitness values was calculated in each partition. Thereafter, the lowest average calculated will be considered to determine the best sub-zone that will be selected for further search process. To confirm that the best sub-zone was chosen, a new procedure was injected to perform a periodic random search throughout all the sub-zones in order to detect the desired sub-zone. This procedure is named as detection procedure in this paper to reflect its main role. Twelve benchmark functions are used to investigate the positive impact of this technique. The results demonstrate the advantages of the anticipated technique over previous improved algorithms, and suggest a new method for optimization search. However, the results also show that the detection procedure decreases the evaluation iterations, which leads to minimize the convergence rate towards the optimal solution. Thus, this paper opens a new dimension for further exploration.