Performance of Sequential Local Algorithms for the Random NAE-$K$-SAT Problem

In this paper, we unify survey propagation (SP) for constraint-satisfaction problems as "probabilistic token passing" algorithms. We show that the reduction of SP from belief propagation (BP) for general problems requires a modification of the BP message-passing rule, namely, introducing what we call a state-decoupling operation on the BP messages. This raises a question mark on the recent "folk belief" that SP is BP.

This paper continued the study of the complexity behavior of the satisfiability analysis of hard random formulas given in the conjunctive normal form (CNF). In the 3-CNF formula, each clause contains three literals of logical variables. The number of logical variables in the formula is N. In this paper, the SAT solver is improved by introducing equality reduction and pure literal identification procedures. The solver improvement reduced the exponent (with base 2) from N/20.86 to N/21.41 with an R=4.6 ratio of the number of clauses to the number of variables. The results show that the efficiency of the pure literals identification procedure decreases as R increases. An important part of the study is an empirical estimation of the algorithmic complexity of the SAT problem with large number of variables. The proposed method gives a convenient lower bound on the complexity of analysis for random 3-CNF formulas. We estimated the algorithmic complexity for the range N=256÷8192. The exponential dependence of complexity on N for random 3-CNF formulas at a fixed value of R is demonstrated in this range.

Linear search algorithms have been shown to be particularly effective for solving partial Maximum Satisfiability (MaxSAT) problem instances. These algorithms start by adding a new relaxation variable to each soft clause and solving the resulting formula with a SAT solver. Whenever a Model is found, a new constraint on the relaxation variables is added such that Models with a greater or equal value are excluded. However, if the problem instance has a large number of relaxation variables, then adding a new constraint over these variables can lead to the exploration of a much larger search space. This article proposes new algorithms to improve the performance of linear search algorithms for MaxSAT by using Models found by the SAT solver to partition the relaxation variables. These algorithms add a new constraint on a subset of relaxation variables, thus intensifying the search on that subspace. In addition, the proposed algorithms are further enhanced by using incremental approaches where learned clauses are kept between calls to the SAT solver. Experimental results show that Model-based algorithms can outperform a classic linear search algorithm in several problem instances. Moreover, incremental approaches further improve the performance of linear search and Model-based algorithms for MaxSAT. Overall, Model-based algorithms are competitive with state-of-the-art MaxSAT solvers, and can improve their performance when solving problem instances with a large number of soft clauses.

The application of Low-Density Generator Matrix (LDGM) Codes combined with Survey Propagation (SP) in steganography seems to be advantageous, since it is possible to approximate the coset leader directly. Thus, large codeword length can be applied, resulting in an embedding efficiency close to the theoretical upper bound of embedding efficiency. Since this approach is still quite complex, this paper deals with the application of Belief Propagation (BP) to LDGM Codes used to reduce the complexity of embedding while keeping the embedding efficiency constant.

In this paper we consider the capacitated vertex cover problem, which is the variant of vertex cover where each node is allowed to cover a limited number of edges. We present an efficient, deterministic, distributed approximation algorithm for the problem. Our algorithm computes a $(2+\epsilon)$-approximate solution which violates the capacity constraints by a factor of $(4+\epsilon)$ in a polylogarithmic number of communication rounds. On the other hand, we also show that every efficient distributed approximation algorithm for this problem must violate the capacity constraints. Our result is achieved in two steps. We first develop a 2-approximate, sequential primal-dual algorithm that violates the capacity constraints by a factor of 2. Subsequently, we present a distributed version of this algorithm. We demonstrate that the sequential algorithm has an inherent need for synchronization which forces any naive distributed implementation to use a linear number of communication rounds. The challenge in this step is therefore to achieve a reduction of the communication complexity to a polylogarithmic number of rounds without worsening the approximation guarantee.

In structured classification problems, there is a direct conflict between expressive models and efficient inference: while graphical models such as Markov random fields or factor graphs can represent arbitrary dependences among instance labels, the cost of inference via belief propagation in these models grows rapidly as the graph structure becomes more complicated. One important source of complexity in belief propagation is the need to marginalize large factors to compute messages. This operation takes time exponential in the number of variables in the factor, and can limit the expressiveness of the models we can use. In this paper, we study a new class of potential functions, which we call decomposable k-way potentials, and provide efficient algorithms for computing messages from these potentials during belief propagation. We believe these new potentials provide a good balance between expressive power and efficient inference in practical structured classification problems. We discuss three instances of decomposable potentials: the associative Markov network potential, the nested junction tree, and a new type of potential which we call the voting potential. We use these potentials to classify images of protein subcellular location patterns in groups of cells. Classifying subcellular location patterns can help us answer many important questions in computational biology, including questions about how various treatments affect the synthesis and behavior of proteins and networks of proteins within a cell. Our new representation and algorithm lead to substantial improvements in both inference speed and classification accuracy.

In this paper, we develop a Belief Propagation (BP) algorithm for similarity computation to improve the recommendation accuracy of the neighborhood method, which is one of the most popular Collaborative Filtering (CF) recommendation algorithms. We formulate a probabilistic inference problem as to compute the marginal posterior distributions of similarity variables from their joint posterior distribution given the observed ratings. However, direct computation is prohibitive in large-scale recommender systems. Therefore, we introduce an appropriate chosen factor graph to express the factorization of the joint distribution function, and utilize the BP algorithm that operates in the factor graph to exploit the factorization for efficient inference. In addition, since the high degree at the factor node incurs an exponential increase in computational complexity, we also propose a complexity-reduction technique. The overall complexity of the proposed BP algorithm on a factor graph is linear in the number of variables, which ensures scalability. Finally, through experiments on the MovieLens dataset, we show the superior prediction accuracy of the proposed BP-based similarity computation algorithm for recommendation.

Disordered and frustrated graphical systems are ubiquitous in physics, biology,and information science. For models on complete graphs or random graphs, deepunderstanding has been achieved through the mean-field replica and cavity methods. Butfinite-dimensional ‘real’ systems remain very challenging because of the abundance of shortloops and strong local correlations. A statistical mechanics theory is constructed in thispaper for finite-dimensional models based on the mathematical framework of the partitionfunction expansion and the concept of region graphs. Rigorous expressions for thefree energy and grand free energy are derived. Message-passing equations on theregion graph, such as belief propagation and survey propagation, are also derivedrigorously.

Preface. 1 Introduction. 1.1 Two examples of combinatorial optimization. 1.2 Why study combinatorial optimization using statistical physics? 1.3 Textbooks. Bibliography. 2 Algorithms. 2.1 Pidgin Algol. 2.2 Iteration and recursion. 2.3 Divide-and-conquer. 2.4 Dynamic programming. 2.5 Backtracking. Bibliography. 3 Introduction to graphs. 3.1 Basic concepts and graph problems. 3.2 Basic graph algorithms. 3.3 Random graphs. Bibliography. 4 Introduction to complexity theory. 4.1 Turing machines. 4.2 Church's thesis. 4.3 Languages. 4.4 The halting problem. 4.5 Class P. 4.6 Class NP. 4.7 Definition of NP-completeness. 4.8 NP-complete problems. 4.9 Worst-case vs. typical-case complexity. Bibliography. 5 Statistical mechanics of the Ising model. 5.1 Phase transitions. 5.2 Some general notes on statistical mechanics. 5.3 The Curie-Weiss model of a ferromagnet. 5.4 The Ising model on a random graph. Bibliography. 6 Algorithms and numerical results for vertex covers. 6.1 Definitions. 6.2 Heuristic algorithms. 6.3 Branch-and-bound algorithm. 6.4 Results: Covering random graphs. 6.5 The leaf-removal algorithm. 6.6 Monte Carlo simulations. 6.7 Backbone. 6.8 Clustering of minimum vertex covers. Bibliography. 7 Statistical mechanics of vertex covers on a random graph. 7.1 Introduction. 7.2 The first-moment bound. 7.3 The hard-core lattice gas. 7.4 Replica approach. Bibliography. 8 The dynamics of vertex-cover algorithms. 8.1 The typical-case solution time of a complete algorithm. 8.2 The dynamics of generalized leaf-removal algorithms. 8.3 Random restart algorithms. Bibliography. 9 Towards new, statistical-mechanics motivated algorithms. 9.1 The cavity graph. 9.2 Warning propagation. 9.3 Belief propagation. 9.4 Survey propagation. 9.5 Numerical experiments on random graphs. Bibliography. 10 The satisfiability problem. 10.1 SAT algorithms. 10.2 Phase transitions in random K-SAT. 10.3 Typical-case dynamics of RandomWalkSAT. 10.4 Message-passing algorithms for SAT. Bibliography. 11 Optimization problems in physics. 11.1 Monte Carlo optimization. 11.2 Hysteric optimization. 11.3 Genetic algorithms. 11.4 Shortest paths and polymers in random media. 11.5 Maximum flows and random-field systems. 11.6 Submodular functions and free energy of Potts model. 11.7 Matchings and spin glasses. Bibliography. Index.

We consider a one-step replica symmetry breaking description of the Edwards–Anderson spin glass model in 2D. The ingredients of this description are a Kikuchi approximation to the free energy and a second-level statistical model built on the extremal points of the Kikuchi approximation, which are also fixed points of a generalized belief propagation (GBP) scheme. We show that a generalized free energy can be constructed where these extremal points are exponentially weighted by their Kikuchi free energy and a Parisi parameter y, and that the Kikuchi approximation of this generalized free energy leads to second-level, one-step replica symmetry breaking (1RSB), GBP equations. We then proceed analogously to the Bethe approximation case for tree-like graphs, where it has been shown that 1RSB belief propagation equations admit a survey propagation solution. We discuss when and how the one-step-replica symmetry breaking GBP equations that we obtain also allow a simpler class of solutions which can be interpreted as a class of generalized survey propagation equations for the single instance graph case.

We introduce an efficient message passing scheme for solving Constraint Satisfaction Problems (CSPs), which uses stochastic perturbation of Belief Propagation (BP) and Survey Propagation (SP) messa...

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT AUTHOR'S REQUEST.] The possibilistic c-means (PCM) was developed as an extension of the fuzzy c-means (FCM) clustering algorithm by abandoning the membership sum-to-one constraint. In the PCM, each cluster is independent of the other clusters and can be processed separately. Because of this separability, the sequential possibilistic one-mean (SP1M) was proposed to find clusters sequentially by running the possibilistic one-mean (P1M) c times. One critical problem in both PCM and SP1M is how to determine the parameter [subscript]. The sequential possibilistic one-mean with adaptive eta (SP1M-AE) was developed to allow [subscript] to change during iterations. In this thesis, a new dynamic adaptation mechanism for the parameter [subscript] in each cluster is inserted into SP1M. The resultant algorithm, called the sequential possibilistic one-mean with dynamic eta (SP1M-DE) is shown to provide superior performance over PCM, SP1M, and SP1M-AE in determining correct clustering results. In this thesis, the family of the SP1M clustering algorithm is also extended to combine local spatial information in an image. The resultant algorithm, called sequential possibilistic local information one-mean (SPLI1M) is shown to have a better performance on image segmentation over the fuzzy local information c-means (FLICM), the possibilistic local information c-means (PLICM), and the family of SP1M without combing local spatial information in image.

We consider the algorithmic task of computing a maximal autarky for a clause-set F, i.e., a partial assignment which satisfies every clause of F it touches, and where this property is destroyed by adding any non-empty set of further assignments. We employ SAT solvers as oracles here, and we are especially concerned with minimising the number of oracle calls. Using the standard SAT oracle, $$\log _2(n(F))$$ oracle calls suffice, where n(F) is the number of variables, but the drawback is that (translated) cardinality constraints are employed, which makes this approach less efficient in practice. Using an extended SAT oracle, motivated by the capabilities of modern SAT solvers, we show how to compute maximal autarkies with $$2 \sqrt{n(F)}$$ simpler oracle calls, by a novel algorithm, which combines the previous two main approaches, based on the autarky-resolution duality and on SAT translations.

Modern CDCL SAT solvers routinely solve very large industrial SAT instances in relatively short periods of time. It is clear that these solvers somehow exploit the structure of real-world instances. However, to-date there have been few results that precisely characterise this structure. In this paper, we provide evidence that the community structure of real-world SAT instances is correlated with the running time of CDCL SAT solvers. It has been known for some time that real-world SAT instances, viewed as graphs, have natural communities in them. A community is a sub-graph of the graph of a SAT instance, such that this sub-graph has more internal edges than outgoing to the rest of the graph. The community structure of a graph is often characterised by a quality metric called Q. Intuitively, a graph with high-quality community structure (high Q) is easily separable into smaller communities, while the one with low Q is not. We provide three results based on empirical data which show that community structure of real-world industrial instances is a better predictor of the running time of CDCL solvers than other commonly considered factors such as variables and clauses. First, we show that there is a strong correlation between the Q value and Literal Block Distance metric of quality of conflict clauses used in clause-deletion policies in Glucose-like solvers. Second, using regression analysis, we show that the the number of communities and the Q value of the graph of real-world SAT instances is more predictive of the running time of CDCL solvers than traditional metrics like number of variables or clauses. Finally, we show that randomly-generated SAT instances with 0.05 ≤ Q ≤ 0.13 are dramatically harder to solve for CDCL solvers than otherwise.

The introduction of vector and parallel supercomputers in the last decade generates a significant impact on large-scale scientific computing. Computational optimization is one important area that is rapidly evolving to benefit from novel computer architectures. In this paper, we restrict our discussions to numerical methods for nonlinear unconstrained optimization. Many important applications fall into this category, and these methods often serve as the basis for other iterative procedures that are generally used in solving nonlinear programming problems. There exists a large array of methods for solving nonlinear unconstrained optimization problems Various implementations of many of these methods can be obtained from the open literature including journals and textbooks. Only a selective number of those techniques have been shown to be most robust and efficient; some of them are currently distributed as commercial software libraries. Testing and comparing these optimization codes is very important since it provides some prior measure of efficiency and robustness of the codes and will show the suitability of a code for solving a particular application problem at hand. Although many numerical evaluations of unconstrained optimization methods have been published, most of them were geared towards relatively small dimensional problems, and very few of them were based on implementations available in public software libraries. Comparison of specific implementations is necessary since the performance of a particular algorithm can vary widely from implementation to implementation. Technical details of the programming and slight changes in the line search techniques, restart criterion, updating formulas, and the like could make a significant difference in the performance of an algorithm. In this paper, we compare the performances of several unconstrained optimization codes available in the leading numerical libraries. These software libraries receive widespread usage in the scientific community worldwide. In particular, we are interested in solving problems for which the number of variables n is of medium to large size, say up to 10,000 variables. Numerical results are obtained using an NEC SX-IA supercomputer.