Towards improved benchmarking of black-box optimization algorithms using clustering problems

This paper analyses the data clustering problem from the continuous black-box optimization point of view and proposes methodological guidelines for a standard benchmark of clustering problem instances. Clustering problems have been used many times in the literature to evaluate evolutionary, metaheuristic and other global optimization algorithms. However much of this work has occurred independently and the various experimental methodologies used have produced results which tend to be incomparable and provide little collective wisdom as to the difficulty of the problems used, or an objective measure for comparing and evaluating the performance of algorithms. This paper surveys some of the clustering literature and results to identify issues relevant for benchmarking. A set of 27 problem instances ranging from 4-D to 40-D and based on three well-known datasets is identified. To establish some pilot results on this benchmark set, experiments are presented for the Covariance Matrix Adaptation-Evolution Strategy and several other standard algorithms. A web-repository has also been created for this problem set to facilitate better experimental evaluations of algorithms.KeywordsAlgorithm BenchmarkingContinuous Black-box OptimizationClustering

We propose a graph model for mutual information based clustering problem. This problem was originally formulated as a constrained optimization problem with respect to the conditional probability distribution of clusters. Based on the stationary distribution induced from the problem setting, we propose a function which measures the relevance among data objects under the problem setting. This function is utilized to capture the relation among data objects, and the entire objects are represented as an edge-weighted graph where pairs of objects are connected with edges with their relevance. We show that, in hard assignment, the clustering problem can be approximated as a combinatorial problem over the proposed graph model when data is uniformly distributed. By representing the data objects as a graph based on our graph model, various graph based algorithms can be utilized to solve the clustering problem over the graph. The proposed approach is evaluated on the text clustering problem over 20 Newsgroup and TREC datasets. The results are encouraging and indicate the effectiveness of our approach.

Fuzzy clustering with structural constraints

Deterministic branch-and-bound algorithms for continuous global optimization often visit a large number of boxes in the neighborhood of a global minimizer, resulting in the so-called cluster problem (Du and Kearfott in J Glob Optim 5(3):253---265, 1994). This article extends previous analyses of the cluster problem in unconstrained global optimization (Du and Kearfott 1994; Wechsung et al. in J Glob Optim 58(3):429---438, 2014) to the constrained setting based on a recently-developed notion of convergence order for convex relaxation-based lower bounding schemes. It is shown that clustering can occur both on nearly-optimal and nearly-feasible regions in the vicinity of a global minimizer. In contrast to the case of unconstrained optimization, where at least second-order convergent schemes of relaxations are required to mitigate the cluster problem when the minimizer sits at a point of differentiability of the objective function, it is shown that first-order convergent lower bounding schemes for constrained problems may mitigate the cluster problem under certain conditions. Additionally, conditions under which second-order convergent lower bounding schemes are sufficient to mitigate the cluster problem around a global minimizer are developed. Conditions on the convergence order prefactor that are sufficient to altogether eliminate the cluster problem are also provided. This analysis reduces to previous analyses of the cluster problem for unconstrained optimization under suitable assumptions.

We study the rank of the Sum of Squares (SoS) hierarchy over the Boolean hypercube for Symmetric Quadratic Functions (SQFs) in $n$ variables with roots placed in points $k-1$ and $k$. Functions of this type have played a central role in deepening the understanding of the performance of the SoS method for various unconstrained Boolean hypercube optimization problems, including the Max Cut problem. Recently, Lee, Prakash, de Wolf, and Yuen proved a lower bound on the SoS rank for SQFs of $\Omega(\sqrt{k(n-k)})$ and conjectured the lower bound of $\Omega(n)$ by similarity to a polynomial representation of the $n$-bit OR function. Using Chebyshev polynomials, we refute the Lee -- Prakash -- de~Wolf -- Yuen conjecture and prove that the SoS rank for SQFs is at most $O(\sqrt{nk}\log(n))$. We connect this result to two constrained Boolean hypercube optimization problems. First, we provide a degree $O( \sqrt{n})$ SoS certificate that matches the known SoS rank lower bound for an instance of Min Knapsack, a problem that was intensively studied in the literature. Second, we study an instance of the Set Cover problem for which Bienstock and Zuckerberg conjectured an SoS rank lower bound of $n/4$. We refute the Bienstock -- Zuckerberg conjecture and provide a degree $O(\sqrt{n}\log(n))$ SoS certificate for this problem.

An important challenge in black-box optimization is to be able to understand the relative performance of different algorithms on problem instances. This challenge has motivated research in exploratory landscape analysis and algorithm selection, leading to a number of frameworks for analysis. However, these procedures often involve significant assumptions, or rely on information not typically available. In this paper we propose a new, model-based framework for the characterization of black-box optimization problems using Gaussian Process regression. The framework allows problem instances to be compared to each other in a relatively simple way. The model-based approach also allows us to assess the goodness of fit and Gaussian Processes lead to an efficient means of model comparison. The implementation of the framework is described and validated on several test sets as one benchmark problem is slowly transformed into another.

A multi-objective improved teaching–learning based optimization algorithm (MO-ITLBO)

Two Stage Surrogate Assisted Framework For Box-Constrained Global Optimisation

The permutation flowshop scheduling problem (PFSP) is one of the most well-known and well-studied production scheduling problems with strong industrial background. This paper presents a new hybrid optimization algorithm which combines the strong global search ability of artificial immune system (AIS) with a strong local search ability of extremal optimization (EO) algorithm. The proposed algorithm is applied to a set of benchmark problems with a makespan criterion. Performance of the algorithm is evaluated. Comparison results indicate that this new method is an effective and competitive approach to the PFSP.

In this work we study the Positive Extension property and Hilbert's 17th problem for real analytic germs and sets. A real analytic germ X0 of has the property if every positive semidefinite analytic function germ on X0 has a positive semidefinite analytic extension to ; analogously one states the property for a global real analytic set X in an open set Ω of ℝn. These properties are natural variations of Hilbert's 17th problem. Here, we prove that: (1) A real analytic germ has the property if and only if every positive semidefinite analytic function germ on X0 is a sum of squares of analytic function germs on X0; and (2) a global real analytic set X of dimension ≦ 2 and local embedding dimension ≦ 3 has the property if and only if it is coherent and all its germs have the property. If that is the case, every positive semidefinite analytic function on X is a sum of squares of analytic functions on X. Moreover, we classify the singularities with the property.

Hippopotamus algorithm for optimal sizing of hybrid renewable energy system

The performance of heuristic algorithms for combinatorial optimization is often highly sensitive to problem instances. A specialized heuristic algorithm may perform exceptionally well on a particular set of instances while fail to produce acceptable solutions on others. This paper proposes a formal method for comparing and selecting heuristic algorithms in real-time given a desired confidence level and a particular set of problem instances. We formulate this algorithm selection problem as a stochastic optimization problem. Two approaches for optimization, ordinal optimization and computing budget allocation, are applied to solve this algorithm selection problem. Computational testing on a set of statistical clustering algorithms in the IMSL library is conducted, which demonstrates that our method can effectively compare and select algorithms that are expected to perform well on given problem instances.

This work investigates robust monotonic convergent iterative learning control (ILC) for uncertain linear systems in both time and frequency domains, and the ILC algorithm optimizing the convergence speed in terms ofl2norm of error signals is derived. First, it is shown that the robust monotonic convergence of the ILC system can be established equivalently by the positive definiteness of a matrix polynomial over some set. Then, a necessary and sufficient condition in the form of sum of squares for the positive definiteness is proposed, which is amendable to the feasibility of linear matrix inequalities. Based on such a condition, the optimal ILC algorithm that maximizes the convergence speed is obtained by solving a set of convex optimization problems. Moreover, the order of the learning function can be chosen arbitrarily so that the designers have the flexibility to decide the complexity of the learning algorithm.

This paper presents a meta-algorithm for approximating the Pareto optimal set of costly black-box multiobjective optimization problems given a limited number of objective function evaluations. The key idea is to switch among different algorithms during the optimization search based on the predicted performance of each algorithm at the time. Algorithm performance is modeled using a machine learning technique based on the available information. The predicted best algorithm is then selected to run for a limited number of evaluations. The proposed approach is tested on several benchmark problems and the results are compared against those obtained using any one of the candidate algorithms alone.

A clustering approach for EOS lumping — Using evolutionary-based metaheuristic optimization algorithms