Perturbed message passing for constraint satisfaction problems

In this paper, we unify survey propagation (SP) for constraint-satisfaction problems as "probabilistic token passing" algorithms. We show that the reduction of SP from belief propagation (BP) for general problems requires a modification of the BP message-passing rule, namely, introducing what we call a state-decoupling operation on the BP messages. This raises a question mark on the recent "folk belief" that SP is BP.

Practical implementation of random binning is one of the key challenges in achieving the largest available rate regions for many multiuser channels. This paper explores the use of low-density parity-check (LDPC) like codes for a particular kind of deterministic broadcast channel called the Blackwell channel and illustrates that random linear codes can be used to construct practical binning schemes at rates close to the capacity region of the Blackwell channel. The key ingredient is an encoding algorithm known as which is a generalization of the well-known belief propagation algorithm for LDPC codes. Survey propagation has been previously devised for a class of constraint satisfaction problems called K-SAT. This paper shows that the encoding problem for the Blackwell channel contains the same features as the constraint satisfaction problem and that the survey propagation algorithm, when concatenated with an outer error correcting code, works well at rates close to the Blackwell channel capacity region

The exponential-time hypothesis (ETH) states that 3-SAT is not solvable in subexponential time, i.e. not solvable in O(c^n) time for arbitrary c > 1, where n denotes the number of variables. Problems like k-SAT can be viewed as special cases of the constraint satisfaction problem (CSP), which is the problem of determining whether a set of constraints is satisfiable. In this paper we study the worst-case time complexity of NP-complete CSPs. Our main interest is in the CSP problem parameterized by a constraint language Gamma (CSP(Gamma)), and how the choice of Gamma affects the time complexity. It is believed that CSP(Gamma) is either tractable or NP-complete, and the algebraic CSP dichotomy conjecture gives a sharp delineation of these two classes based on algebraic properties of constraint languages. Under this conjecture and the ETH, we first rule out the existence of subexponential algorithms for finite domain NP-complete CSP(Gamma) problems. This result also extends to certain infinite-domain CSPs and structurally restricted CSP(Gamma) problems. We then begin a study of the complexity of NP-complete CSPs where one is allowed to arbitrarily restrict the values of individual variables, which is a very well-studied subclass of CSPs. For such CSPs with finite domain D, we identify a relation SD such that (1) CSP({SD}) is NP-complete and (2) if CSP(Gamma) over D is NP-complete and solvable in O(c^n) time, then CSP({SD}) is solvable in O(c^n) time, too. Hence, the time complexity of CSP({SD}) is a lower bound for all CSPs of this particular kind. We also prove that the complexity of CSP({SD}) is decreasing when |D| increases, unless the ETH is false. This implies, for instance, that for every c>1 there exists a finite-domain Gamma such that CSP(Gamma) is NP complete and solvable in O(c^n) time.

The application of Low-Density Generator Matrix (LDGM) Codes combined with Survey Propagation (SP) in steganography seems to be advantageous, since it is possible to approximate the coset leader directly. Thus, large codeword length can be applied, resulting in an embedding efficiency close to the theoretical upper bound of embedding efficiency. Since this approach is still quite complex, this paper deals with the application of Belief Propagation (BP) to LDGM Codes used to reduce the complexity of embedding while keeping the embedding efficiency constant.

Constraint satisfaction problems usually exhibit a distinct satisfiability transition phenomenon under stochastic models, and an exact phase transition threshold has been strictly proven to exist especially in the RB model. Aiming at the difficulty in solving hard instances of the random binary model with growing domains (RB) near the phase transition region, this paper proposes a guided decimation algorithm based on asynchronous belief propagation with gradual perturbation (P-BP). On the basis of the asynchronous BP-guided decimation framework, the algorithm introduces a linear annealing perturbation mechanism, which enables the variable-to-constraint messages to transit smoothly from deterministic update to the style of Gibbs sampling. Meanwhile, in the late stage of decimation, the variable fixing strategy is changed from greedy selection to direct sampling from the marginal distribution, supplemented by an automatic restart strategy with a maximum of 3 restarts. These improvements retain the asynchronous update, damping factor and A/B/C edge processing rules of the original algorithm, while significantly enhancing the stochastic exploration capability and effectively alleviating the problems of message oscillation, local convergence and error propagation of fixed variables in the phase transition region. Research shows that the combination of gradual perturbation with late-stage probabilistic sampling and a restart mechanism provides an effective way to enhance the robustness of belief propagation-based algorithms for stochastic constraint satisfaction problems.

The Constraint Satisfaction Problem (CSP) is a central and generic computational problem which provides a common framework for many theoretical and practical applications. A central line of research is concerned with the identification of classes of instances for which CSP can be solved in polynomial time; such classes are often called “islands of tractability.” A prominent way of defining islands of tractability for CSP is to restrict the relations that may occur in the constraints to a fixed set, called a constraint language , whereas a constraint language is conservative if it contains all unary relations. Schaefer’s famous Dichotomy Theorem (STOC 1978) identifies all islands of tractability in terms of tractable constraint languages over a Boolean domain of values. Since then, many extensions and generalizations of this result have been obtained. Recently, Bulatov (TOCL 2011, JACM 2013) gave a full characterization of all islands of tractability for CSP and the counting version #CSP that are defined in terms of conservative constraint languages. This article addresses the general limit of the mentioned tractability results for CSP and #CSP, that they only apply to instances where all constraints belong to a single tractable language (in general, the union of two tractable languages is not tractable). We show that we can overcome this limitation as long as we keep some control of how constraints over the various considered tractable languages interact with each other. For this purpose, we utilize the notion of a strong backdoor of a CSP instance, as introduced by Williams et al. (IJCAI 2003), which is a set of variables that when instantiated, moves the instance to an island of tractability, that is, to a tractable class of instances. We consider strong backdoors into scattered classes , consisting of CSP instances where each connected component belongs entirely to some class from a list of tractable classes. Figuratively speaking, a scattered class constitutes an archipelago of tractability . The main difficulty lies in finding a strong backdoor of given size k ; once it is found, we can try all possible instantiations of the backdoor variables and apply the polynomial time algorithms associated with the islands of tractability on the list component-wise. Our main result is an algorithm that, given a CSP instance with n variables, finds in time f ( k ) n O (1) a strong backdoor into a scattered class (associated with a list of finite conservative constraint languages) of size k or correctly decides that there is not such a backdoor. This also gives the running time for solving (#)CSP, provided that (#)CSP is polynomial-time tractable for the considered constraint languages. Our result makes significant progress towards the main goal of the backdoor-based approach to CSPs—the identification of maximal base classes for which small backdoors can be detected efficiently.

We formalize the class of “sequential local and show that these algorithms fail to find satisfying assignments on random instances of the “Not-All-Equal-$K$-SAT” (NAE-$K$-SAT) problem if the number of message passing iterations is bounded by a function moderately growing in the number of variables and if the clause-to-variable ratio is above $(1+o_K(1)){2^{K-1}\over K}\ln^2 K$ for sufficiently large $K$. Sequential local algorithms are those that iteratively set variables based on some local information and/or local randomness and then recurse on the reduced instance. Our model captures some weak abstractions of natural algorithms such as Survey Propagation (SP)-guided as well as Belief Propagation (BP)-guided decimation algorithms---two widely studied message-passing--based algorithms---when the number of message-passing rounds in these algorithms is restricted to be growing only moderately with the number of variables. The approach underlying our paper is based on an intricate geometry of th...

Disordered and frustrated graphical systems are ubiquitous in physics, biology,and information science. For models on complete graphs or random graphs, deepunderstanding has been achieved through the mean-field replica and cavity methods. Butfinite-dimensional ‘real’ systems remain very challenging because of the abundance of shortloops and strong local correlations. A statistical mechanics theory is constructed in thispaper for finite-dimensional models based on the mathematical framework of the partitionfunction expansion and the concept of region graphs. Rigorous expressions for thefree energy and grand free energy are derived. Message-passing equations on theregion graph, such as belief propagation and survey propagation, are also derivedrigorously.

Preface. 1 Introduction. 1.1 Two examples of combinatorial optimization. 1.2 Why study combinatorial optimization using statistical physics? 1.3 Textbooks. Bibliography. 2 Algorithms. 2.1 Pidgin Algol. 2.2 Iteration and recursion. 2.3 Divide-and-conquer. 2.4 Dynamic programming. 2.5 Backtracking. Bibliography. 3 Introduction to graphs. 3.1 Basic concepts and graph problems. 3.2 Basic graph algorithms. 3.3 Random graphs. Bibliography. 4 Introduction to complexity theory. 4.1 Turing machines. 4.2 Church's thesis. 4.3 Languages. 4.4 The halting problem. 4.5 Class P. 4.6 Class NP. 4.7 Definition of NP-completeness. 4.8 NP-complete problems. 4.9 Worst-case vs. typical-case complexity. Bibliography. 5 Statistical mechanics of the Ising model. 5.1 Phase transitions. 5.2 Some general notes on statistical mechanics. 5.3 The Curie-Weiss model of a ferromagnet. 5.4 The Ising model on a random graph. Bibliography. 6 Algorithms and numerical results for vertex covers. 6.1 Definitions. 6.2 Heuristic algorithms. 6.3 Branch-and-bound algorithm. 6.4 Results: Covering random graphs. 6.5 The leaf-removal algorithm. 6.6 Monte Carlo simulations. 6.7 Backbone. 6.8 Clustering of minimum vertex covers. Bibliography. 7 Statistical mechanics of vertex covers on a random graph. 7.1 Introduction. 7.2 The first-moment bound. 7.3 The hard-core lattice gas. 7.4 Replica approach. Bibliography. 8 The dynamics of vertex-cover algorithms. 8.1 The typical-case solution time of a complete algorithm. 8.2 The dynamics of generalized leaf-removal algorithms. 8.3 Random restart algorithms. Bibliography. 9 Towards new, statistical-mechanics motivated algorithms. 9.1 The cavity graph. 9.2 Warning propagation. 9.3 Belief propagation. 9.4 Survey propagation. 9.5 Numerical experiments on random graphs. Bibliography. 10 The satisfiability problem. 10.1 SAT algorithms. 10.2 Phase transitions in random K-SAT. 10.3 Typical-case dynamics of RandomWalkSAT. 10.4 Message-passing algorithms for SAT. Bibliography. 11 Optimization problems in physics. 11.1 Monte Carlo optimization. 11.2 Hysteric optimization. 11.3 Genetic algorithms. 11.4 Shortest paths and polymers in random media. 11.5 Maximum flows and random-field systems. 11.6 Submodular functions and free energy of Potts model. 11.7 Matchings and spin glasses. Bibliography. Index.

We consider a one-step replica symmetry breaking description of the Edwards–Anderson spin glass model in 2D. The ingredients of this description are a Kikuchi approximation to the free energy and a second-level statistical model built on the extremal points of the Kikuchi approximation, which are also fixed points of a generalized belief propagation (GBP) scheme. We show that a generalized free energy can be constructed where these extremal points are exponentially weighted by their Kikuchi free energy and a Parisi parameter y, and that the Kikuchi approximation of this generalized free energy leads to second-level, one-step replica symmetry breaking (1RSB), GBP equations. We then proceed analogously to the Bethe approximation case for tree-like graphs, where it has been shown that 1RSB belief propagation equations admit a survey propagation solution. We discuss when and how the one-step-replica symmetry breaking GBP equations that we obtain also allow a simpler class of solutions which can be interpreted as a class of generalized survey propagation equations for the single instance graph case.

Handbook of Constraint Programming

Conventional techniques for the constraint satisfaction problem (CSP) have had considerable success in their applications. However, there are many areas in which the performance of the basic approaches may be improved. These include heuristic ordering of certain tasks performed by the CSP solver, hybrids which combine compatible solution techniques and graph based methods which exploit the structure of the constraint graph representation of a CSP. Also, conventional constraint satisfaction techniques only address problems with hard constraints (i.e. each of which are completely satisfied or completely violated, and all of which must be satisfied by a valid solution). Many real applications require a more flexible approach which relaxes somewhat these rigid requirements. To address these issues various approaches have been developed. This paper attempts a systematic review of them.

An Iterative local-search framework for solving constraint satisfaction problem

We discuss a coevolutionary genetic algorithm for constraint satisfaction. Our basic idea is to explore effective genetic information in the population, i.e., schemata, and to exploit the genetic information in order to guide the population to better solutions. Our coevolutionary genetic algorithm (CGA) consists of two GA populations; the first GA, called searches for the solutions in a given environment (problem), and the second GA, called P-GA, searches for effective genetic information involved in the H-GA, namely, good schemata. Thus, each individual in P-GA consists of alleles in H-GA or don't care symbol representing a schema in the H-GA. These GA populations separately evolve in each genetic space at different abstraction levels and affect with each other by two genetic operators: superposition and transcription. We then applied our CGA to constraint satisfaction problems (CSPs) incorporating a new stochastic repair operator for P-GA to raise the consistency of schemata with the (local) constraint conditions in CSPs. We carried out two experiments: First, we examined the performance of CGA on various general CSPs that are generated randomly for a wide variety of density and tightness of constraint conditions in the CSPs that are the basic measures of characterizing CSPs. Next, we examined structured CSPs involving latent cluster structures among the variables in the CSPs. For these experiments, computer simulations confirmed us the effectiveness of our CGA.

This paper presents a VLSI design for a competitive neural network model, known as GENET (Wang and Tsang 1991), for solving Constraint Satisfaction Problems (CSP). The CSP is a mathematical abstraction of the problems in many AI application domains. In essence, a CSP can be defined as a triple (Z, D, C), where Z is a finite set of variables, D is a mapping from every variable to a domain, which is a finite set of arbitrary objects, and C is a set of constraints. Each constraint in C restricts the values that can be simultaneously assigned to a number of variables in Z. If the constraints in C involve up to but no more than n variables it is called an n-ary CSP. The task is to assign one value per variable satisfying all the constraints in C (Mackworth 1977). In addition, associated with the variable assignments might be costs and utilities. This turns CSPs into optimization problems, demanding CSP solvers to find a set of variable assignments that would produce a maximum total utility at a minimal cost. Furthermore, some CSPs might be over-constrained, i.e. not all the constraints in C can be satisfied simultaneously. In this case, the set of assignments to a maximum number of variables without violating any constraints in C, or the set of assignments to all the variables which violates a minimal number of constraints might be sought for.