Learning MAX-SAT from contextual examples for combinatorial optimisation

The main objective of this paper is to provide a state-of-the-art review, analyze and discuss stochastic local search techniques used for solving hard combinatorial problems. It begins with a short introduction, motivation and some basic notation on combinatorial problems, search paradigms and other relevant features of searching techniques as needed for background. In the following a brief overview of the stochastic local search methods along with an analysis of the state-of-the-art stochastic local search algorithms is given. Finally, the last part of the paper present and discuss some of the most latest trends in application of stochastic local search algorithms in machine learning, data mining and some other areas of science and engineering. We conclude with a discussion on capabilities and limitations of stochastic local search algorithms.

Contributors

Stochastic local search (SLS), like many other stochastic optimization algorithms, has several parameters that need to be optimized in order for the algorithm to find high quality solutions within a short amount of time. In this paper, we formulate a stochastic local search bandit (\(\mathtt{SLSB}\)), which is a novel learning variant of SLS based on multi-armed bandits. \(\mathtt{SLSB}\) optimizes SLS over a sequence of stochastic optimization problems and achieves high average cumulative reward. In \(\mathtt{SLSB}\), we study how SLS can be optimized via low degree polynomials in its noise and restart parameters. To determine the coefficients of the polynomials, we present polynomial Thompson Sampling (\(\mathtt{PolyTS}\)). We derive a regret bound for \(\mathtt{PolyTS}\) and validate its performance on synthetic problems of varying difficulty as well as on feature selection problems. Compared to bandits with no assumptions of the reward function and other parameter optimization approaches, our \(\mathtt{PolyTS}\) assuming polynomial structure can provide substantially better parameter optimization for SLS. In addition, due to its simple model update, \(\mathtt{PolyTS}\) has low computational cost compared to other SLS parameter optimization methods.

In This Issue

Feature selection is a challenging task that has been the subject of a large amount of research, especially in relation to classification tasks. It permits to eliminate the redundant attributes and enhance the classification accuracy by keeping only the relevant attributes. In this paper, we propose a hybrid search method based on both harmony search algorithm (HSA) and stochastic local search (SLS) for feature selection in data classification. A novel probabilistic selection strategy is used in HSA---SLS to select the appropriate solutions to undergo stochastic local refinement, keeping a good compromise between exploration and exploitation. In addition, the HSA---SLS is combined with a support vector machine (SVM) classifier with optimized parameters. The proposed HSA---SLS method tries to find a subset of features that maximizes the classification accuracy rate of SVM. Experimental results show good performance in favor of our proposed method.

Choosing a suitable feature representation for structured data is a non-trivial task due to the vast number of potential candidates. Ideally, one would like to pick a small, but informative set of structural features, each providing complementary information about the instances. We frame the search for a suitable feature set as a combinatorial optimization problem. For this purpose, we define a scoring function that favors features that are as dissimilar as possible to all other features. The score is used in a stochastic local search (SLS) procedure to maximize the diversity of a feature set. In experiments on small molecule data, we investigate the effectiveness of a forward selection approach with two different linear classification schemes.KeywordsCombinatorial Optimization ProblemHadamard MatriceTraining AccuracyStochastic Local SearchTraining MatrixThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

MAX-SAT, the optimisation variant of the satisfiability problem in propositional logic, is an important and widely studied combinatorial optimisation problem with applications in AI and other areas of computing science. In this paper, we present a new stochastic local search (SLS) algorithm for MAX-SAT that combines Iterated Local Search and Tabu Search, two well-known SLS methods that have been successfully applied to many other combinatorial optimisation problems. The performance of our new algorithm exceeds that of current state-of-the-art MAX-SAT algorithms on various widely studied classes of unweighted And weighted MAX-SAT instances, particularly for Random-3-SAT instances with high variance clause weight distributions. We also report promising results for various classes of structured MAX-SAT instances.

Real-world problems often have parameters that are uncertain during the optimization phase; stochastic optimization or stochastic programming is a key approach introduced by Beale and by Dantzig in the 1950s to address such uncertainty. Matching is a classical problem in combinatorial optimization. Modern stochastic versions of this problem model problems in kidney exchange, for instance. We improve upon the current-best approximation bound of 3.709 for stochastic matching due to Adamczyk et al. (in: Algorithms-ESA 2015, Springer, Berlin, 2015) to 3.224; we also present improvements on Bansal et al. (Algorithmica 63(4):733–762, 2012) for hypergraph matching and for relaxed versions of the problem. These results are obtained by improved analyses and/or algorithms for rounding linear-programming relaxations of these problems.

Many stochastic local search (SLS) methods rely on the manipulation of single solutions at each of the search steps. Examples are iterative improvement, iterated local search, simulated annealing, variable neighborhood search, and iterated greedy. These SLS methods are the basis of many state-of-the-art algorithms for hard combinatorial optimization problems. Often, several of these SLS methods are combined with each other to improve performance. We propose here a practical, unified structure that encompasses several such SLS methods. The proposed structure is unified because it integrates these metaheuristics into a single structure from which we can not only instantiate each of them, but we also can generate complex combinations and variants. Moreover, the structure is practical since we propose a method to instantiate actual algorithms for practical problems in a semi-automatic fashion. The method presented in this work implements a general local search structure as a grammar; an instantiation of such a grammar is a program that can be compiled into executable form. We propose to find the appropriate grammar instantiation for a particular problem by means of automatic configuration. The result is a semi-automatic system that, with little human effort, is able to generate powerful hybrid SLS algorithms.KeywordsStochastic local searchgeneralized local search structuregrammarautomatic algorithm design

In the last two decades, computational enhancements highly contributed to the increase in popularity of DTI algorithms. This cause the successful use of Decision Tree Induction (DTI) using recursive partitioning algorithms in many diverse areas such as radar signal classification, character recognition, remote sensing, medical diagnosis, expert systems, and speech recognition, to name only a few. But recursive partitioning and DTI are two faces of

Propositional satisfiability (SAT) is a prominent problem in artificial intelligence with many important applications. Stochastic local search (SLS) is a well-known approach for solving SAT and known to achieve excellent performance on randomly generated, satisfiable instances. However, SLS solvers for SAT are usually ineffective in solving application instances. Here, we propose a highly configurable SLS solver dubbed PbO-CCSAT, which leverages a powerful technique known as configuration checking (CC) in combination with the automatic algorithm design paradigm of programming by optimisation (PbO). Our PbO-CCSAT solver exposes a large number of design choices, which are automatically configured to optimise the performance for specific classes of SAT instances. We present extensive empirical results showing that our PbO-CCSAT solver significantly outperforms state-of-the-art SLS solvers on SAT instances from many applications, and further show that PbO-CCSAT is complementary to state-of-the-art complete solvers.

Improving the performance of stochastic local search for maximum vertex weight clique problem using programming by optimization

Dual-mode dynamics neural network for combinatorial optimization

The particle swarm optimizer (PSO) is a swarm intelligence based heuristic optimization technique that can be applied to a wide range of problems. After analyzing the dynamics of tranditioal PSO, this paper presents a new PSO variant based on local stochastic search strategy (LSSPSO) for performance enhancement. This is inspired by a social phenomenon that everyone wants to first exceed the nearest superior and then all superior. Specifically, LSSPSO adopts a local stochastic search to adjust inertia weight in terms of keeping a balance between the diversity and the convergence speed, aiming to improve the performance of tranditioal PSO. Experiments conducted on unimodal and multimodal test functions demonstrate the effectiveness of LSSPSO in solving multiple benchmark problems as compared to several other PSO variants.Keywordsparticle swarm optimizationcontinuous optimizationlocal stochastic search

In this article, we describe the steps that have been followed in the development of a high performing stochastic local search algorithm for the probabilistic travelling salesman problem, a paradigmatic combinatorial stochastic optimization problem. In fact, we have followed a bottom-up algorithm engineering process that starts from basic algorithms (here, iterative improvement) and adds complexity step-by-step. An extensive experimental campaign has given insight into the advantages and disadvantages of the prototype algorithms obtained at the various steps and directed the further algorithm development. The final stochastic local search algorithm was shown to substantially outperform the previous best algorithms known for this problem. Besides the systematic engineering process for the development of stochastic local search algorithms followed here, the main reason for the high performance of our final algorithm is the innovative adoption of techniques for the estimation of the cost of neighboring solutions using delta evaluation.