Application of a New Bio-Inspired Algorithm in Optimization Problem

Application of a Derivative-Free Method with Projection Skill to Solve an Optimization Problem

We have developed a new derivative-free algorithm based on Radial Basis Functions (RBFs). Derivative-free optimization is an active field of research and several algorithms have been proposed recently. Problems of this nature in the industrial setting are quite frequent. The reason is that in a number of applications the optimization process contains simulation packages which are treated as black boxes. The development of our own algorithm was originally motivated by an application in biomedical imaging: the medical image registration problem. The particular characteristics of this problem have incited us to develop a new optimization algorithm based on trust-region methods. However it has been designed to be generic and to be applied to a wide range of problems. The main originality of our approach is the use of RBFs to build the models. In particular we have adapted the existing theory based on quadratic models to our own models and developed new procedures especially designed for models based on RBFs. We have tested our algorithm called BOOSTERS against state-of-the-art methods (UOBYQA, NEWUOA, DFO). On the medical image registration problem, BOOSTERS appears to be the method of choice. The tests on problems from the CUTEr collection show that BOOSTERS is comparable to, but not better than other methods on small problems (size 2-20). It is performing very well for medium size problems (20-80). Moreover, it is able to solve problems of dimension 200, which is considered very large in derivative-free optimization. We have also developed a new class of algorithms combining the robustness of derivative-free algorithms with the faster rate of convergence characterizing Newtonlike-methods. In fact, they define a new class of algorithms lying between derivative-free optimization and quasi-Newton methods. These algorithms are built on the skeleton of our derivative-free algorithm but they can incorporate the gradient when it is available. They can be interpreted as a way of doping derivative-free algorithms with derivatives. If the derivatives are available at each iteration, then our method can be seen as an alternative to quasi-Newton methods. At the opposite, if the derivatives are never evaluated, then the algorithm is totally similar to BOOSTERS. It is a very interesting alternative to existing methods for problems whose objective function is expensive to evaluate and when the derivatives are not available. In this situation, the gradient can be approximated by finite differences and its costs corresponds to n additional function evaluations assuming that Rn is the domain of definition of the objective function. We have compared our method with CFSQP and BTRA, two gradient-based algorithms, and the results show that our doped method performs best. We have also a theoretical analysis of the medical image registration problem based on maximization of mutual information. Most of the current research in this field is concentrated on registration based on nonlinear image transformation. However, little attention has been paid to the theoretical properties of the optimization problem. In our analysis, we focus on the continuity and the differentiability of the objective function. We show in particular that performing a registration without extension of the reference image may lead to discontinuities in the objective function. But we demonstrate that, under some mild assumptions, the function is differentiable almost everywhere. Our analysis is important from an optimization point of view and conditions the choice of a solver. The usual practice is to use generic optimization packages without worrying about the differentiability of the objective function. But the use of gradient-based methods when the objective function is not differentiable may result in poor performance or even in absence of convergence. One of our objectives with this analysis is also that practitioners become aware of these problems and to propose them new algorithms having a potential interest for their applications.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

In this thesis we analyze and develop methods based on model order reduction (MOR) for the solution of optimization problems constrained by time-dependent partial differential equations (PDEs). The methods combine a direct solution approach with model reduction via proper orthogonal decomposition (POD) and the discrete empirical interpolation method (DEIM). The reduced-order models (ROMs) are used to approximate the high-dimensional dynamic systems originating from a spatial discretization of a PDE. However, when used in an optimization algorithm, conventional POD/DEIM ROMs often lack the ability to give adequate approximations of the gradient. We propose methods for a suitable enhancement of the ROMs for the optimization purpose which are based on the inclusion of derivative information in the POD and DEIM subspaces. We distinguish two types of error between quantities evaluated with the high-dimensional model and its ROM approximation in dependency on the optimization variable q: The reconstruction error which is evaluated with the same q0 which is used constructing the ROM and the prediction error which assesses approximations at q with a ROM constructed at q0 different to q. The novel reconstruction results we present include estimates for solutions of the adjoint equation and the sensitivity equations as well as for the gradient of the objective function. Based on the estimates we explain how the POD and DEIM bases should be extended with either adjoint or sensitivity information. The enhanced ROMs allow control of the reconstruction error for the objective and its gradient up to machine precision. Moreover, we propose a POD prediction estimate for the objective of the optimization problem in a neighborhood of q where the ROM is constructed. In case of sensitivity-extended POD and DEIM bases we give an analogous result for solutions of the states. The derivative-extended ROMs are then used to develop adaptive algorithms for the solution of optimal control and parameter estimation problems which results in great runtime improvements for the optimization while ensuring high approximation quality of the solution of the original problem. For the parameter estimation case a novel a posteriori error estimate is proposed which assesses the quality of suboptimal solutions obtained with the ROM. A further fundamental contribution is a discussion of discretize-then-optimize (DTO) vs. optimize-then-discretize (OTD) approaches in the context of MOR for optimization. We analyze advantages and disadvantages of both approaches and discuss to which extent our methods exhibit properties of either strategy. We also give examples of representative optimization problems in which standard POD/DEIM ROMs show an inacceptable behavior and can be successfully solved by derivative-extended ROMs. We have further implemented the developed methods emphasizing an efficient realization which is important for the investigation of the MOR potential. We showcase the practical performance of the proposed algorithms and the superiority of derivative-extended over conventional ROMs on two academic and one industry-relevant application which exhibit a variety of challenges for the model reduction approach in optimization.

This thesis proposes a new parallel computing network--a binary relation inference network. The network topology is set up to match conventional optimization procedures. The network transforms some optimization problems (shortest path, transitive closure, assignment, etc.) into network convergence problems and divides the work load evenly among all processors. Link weights are known. It can operate in synchronous discrete-time, asynchronous discrete-time, and continuous-time domains. Inference network algorithms are derived directly from problem formulation, and are effective for a large range of problem sizes. The inference network consists of interconnected components; each component is composed of a unit, a number of sites attached to the unit, and links from the unit to sites on other units. Each unit is a simple computational engine. The topology of the inference network matches naturally to many optimization problems, since its deduction sites produce candidates to compete for the optimum and its decision-making units select the optimum from the site values. Either directly or through a transformation mapping to a systolic structure, discrete inference network algorithms can be implemented on a commercially available parallel processing facility, such as the Connection Machine. Analog inference network can be built using integrated circuits which solve problems efficiently in the continuous-time domain. In this thesis, mathematical analysis and numerical simulation were conducted for the synchronous, asynchronous, and analog inference network. Various applications has been discussed. The inference network has shown to solve the all-pair shortest path problem efficiently in various updating schemes. The inference network algorithm for the transitive closure problem parts derived straightforwardly from the formulation of the problem and the topology of the inference network. Efficient continuous-time solution was obtained from a non-synchronized logical circuit. The convergence of the network for the shortest path or transitive closure problems was shown to be independent of the problem size. It was demonstrated that the inference network can solve the assignment problem in a way similar to the Hopfield net's solution to the traveling salesman problem. However, the former was shown to converge for all test cases with problem size up to 128. (Abstract shortened by UMI.)

Since the first communication systems were developed, the scientific community has been witnessing attempts to increase the amount of information that can be transmitted. In the last 10--15 years there has been a tremendous amount of research towards developing multi-antenna systems which would hopefully provide high-data-rate transmissions. However, increasing the overall amount of transmitted information increases the complexity of the necessary signal processing. A large portion of this thesis deals with several important issues in signal processing of multi-antenna systems. In almost every particular case the goal is to develop a technique/algorithm so that the overall complexity of the signal processing is significantly decreased. In the first part of the thesis a very important problem of signal detection in MIMO (multiple-input multiple-output) systems is considered. The problem is analyzed in two different scenarios: when the transmission medium (channel) 1) is known and 2) is unknown at the receiver. The former case is often called coherent and the later non-coherent MIMO detection. Both cases usually amount to solving highly complex NP-hard combinatorial optimization problems. For the coherent case we develop a significant improvement of the traditional sphere decoder algorithm commonly used for this type of detection. An interesting connection between the new improved algorithm and the H-infinity estimation theory is established, and the performance improvement over the standard sphere decoder is demonstrated. For the non-coherent case we develop a counterpart to the standard sphere decoder, the so-called out-sphere decoder. The complexity of the algorithm is viewed as a random variable; its expected value is analyzed and shown to be significantly smaller than the one of the overall exhaustive search. In the non-coherent case, in addition to the complexity analysis of the exact out-sphere decoder, we analyze the performance loss of a suboptimal technique. We show that only a moderate loss of a few dbs in power required at the transmitter will occur if a polynomial algorithm based on the semi-definite relaxation is used in place of any exact technique (which of course is not known to be polynomial). In the second part of the thesis we consider a few problems that arise in wireless broadcast channels. Namely, we consider the problem of the information symbol vector design at the transmitter. A polynomial linear precoding technique is constructed. It enables achieving data rates very close to the ones achieved with DPC (dirty paper coding) technique. Additionally, for another suboptimal polynomial scheme (the so-called nulling and cancelling), we show that it asymptotically achieves the same data rate as the optimal, exponentially complex, DPC. In the last part of the thesis we consider a quantum counterpart of the signal detection from classical communication. In quantum systems the signals are quantum states and the quantum detection problem amounts to designing measurement operators which have to satisfy certain quantum mechanics laws. A specific type of quantum detection called unambiguous detection, which has numerous applications including quantum filtering, has recently attracted a lot of attention in the research community. We develop a general framework for numerically solving this problem using the tools from the convex optimization theory. Furthermore, in the special case where the two quantum states are of rank 2, we construct an explicit analytical solution for the measurement operators. At the end we would like to emphasize that the contribution of this thesis goes beyond the specific problems mentioned here. Most algorithmic optimization techniques developed in this paper are generally applicable. While it is a fact that our results were originally motivated by wireless and quantum communications applications, we believe that the developed techniques will find applications in many different areas where similar optimization problems appear.

In this thesis, we study the theoretical approach on energy-efficient scheduling problems arising in demand response management in the modern electrical smart grid. Consumers send in power requests with flexible feasible timeslots during which their requests can be served. The grid controller, upon receiving power requests, schedules each request within the specified interval. The electricity cost is measured by a convex function of the load in each timeslot. The objective is to schedule all requests with the minimum total electricity cost. We study the smart grid scheduling problem in different models. For the offline model, we prove the problem is NP-hard for the general case. We propose a polynomial time algorithm for special input where jobs have unit power request and unit time duration. By adapting the polynomial time algorithm for unit-size jobs, we propose an approximation algorithm for more general input. On the other hand, we also present an exact algorithm to find the optimal schedule for the problem with general input. For the online model, we propose an online algorithm for jobs with jobs with arbitrary power request, arbitrary time duration, and arbitrary contiguous feasible intervals. We also show a lower bound of the competitive ratio for the smart grid scheduling problem with unit height and arbitrary width. For special cases, we design different online algorithms with better competitive ratios. Finally, we look at other optimization problems and show how to solve them by adapting our techniques. We prove that our online algorithm can solve the machine minimization problem with an asymptotically optimal competitive ratio. We also show that our exact algorithm can be adapted to solve other demand response management problems.

The purpose of this work is to investigate some of the algorithms used for representing switching functions by space efficient decision diagrams. This representation is an optimization problem. Adopting the terminology from combinatorial optimization problems, we have:Problem Instance: A multivalued function f(x) where x is a vector, and a storing cost, s(xi), associated with each variable xi in x.Feasible Solution: A decision diagram D (acyclic, directed graph with indegrees of the internal nodes possibly exeeding one) representing f(x).Objective Function: The total storage cost,S(D) = Σ s(xi) for each feasible solution D.xie DOptimal Solution: A decision diagram D representing the function with minimum total storage cost S(D), i.e. a feasible solution which minimizes the objective function.To date, only two algorithms for the construction of optimal decision diagrams with respect to the total storage cost have been developed; Reinwald & Soland's branch & bound algorithm and Thayse's, based on the extension of Shannon's expansion formula. The second algorithm is analyzed and proven to be highly exponential in the number of variables. This comes as no surprise since, in most cases, the construction of optimal decision trees is an NP-hard problem. The construction of decision diagrams is generally more complicated than that of trees. One cannot seriously doubt that the constructions of optimal diagrams is also NP-hard for most costs including the total storage cost. Therefore, a heuristic approach to this problem is needed. Two heuristics, based on the greedy algorithmic technique, are presented. These heuristics with the analysis of Thayse's algorithm constitute the major achievements of this work.

Optimization problems that involve discrete variables are exposed to the conflict between being a powerful modeling tool and often being hard to solve. Infinite-dimensional processes, as e.g. described by differential equations, underlying the optimization may lead to the need to solve for distributed discrete control variables. This work analyzes approximation arguments that replace the need for solving the optimization problem by the need for first solving a relaxation and second computing appropriate roundings to regain discrete controls. We provide sufficient conditions on rounding algorithms and their grid refinement strategies that allow to prove approximation of the relaxed controls by the discrete controls in weaker topologies, a feature due to the infinite-dimensional vantage point. If the control-to-state mapping of the underlying process exhibits suitable compactness properties, state vector approximation follows in the norm topology as well as, under additional assumptions, optimality principles of the computed discrete controls. The conditions are verified for representatives of the family of Sum-Up Rounding algorithms. We apply the arguments on different classes of mixed-integer optimization problems that are constrained by partial differential equations. Specifically, we consider discrete control inputs, which are distributed in the time domain, for evolution equations that are governed by a differential operator that generates a strongly continuous semigroup, discrete control inputs, which are distributed in multi-dimensional spatial domains, for elliptic boundary value problems and discrete control inputs, which are distributed in space-time cylinders, for evolution equations that are governed by differential operators such that the corresponding Cauchy problem satisfies maximal parabolic regularity. Furthermore, we apply the arguments outside the scope of partial differential equations to a signal reconstruction problem. Computational results illustrate the findings.

This thesis treats different aspects of the class of Mixed-Integer Optimal Control Problems (MIOCPs). These are optimization problems that combine the difficulties of underlying dynamic processes with combinatorial decisions. Typically, these combinatorial decisions are realized as switching decisions between the system’s different operations modes. During the last decades, direct methods emerged as the state-of-the-art solvers for MIOCPs. The formulation of a valid, tight and dependable integral relaxation, i.e., the formulation of a model for fractional values, plays an important role for these direct solution methods. We give detailed insight into several relaxation approaches for MIOCPs and compare them with regard to their respective structures. In particular, these are the typical solution’s structures and properties as convexity, problem size and numerical behavior. From these structural properties, we deduce some required specifications of a solver. Additionally, the modeling and subsequent limitation of the switching process directly tackle the class-specific typical issue of chattering solutions. One of the relaxation methods for MIOCPs is the outer convexification, where the binary variables only enter affinely. For the approximation of this relaxation’s solution, we took up on the control approximation problem in integral sense derived by Sager as part of a decomposition approach for MIOCPs with affine binary controls. This problem describes the optimal approximation of fractional controls with binary controls such that the corresponding dynamic process is changed as little as possible. For the multi-dimensional problem, we developed a new heuristic, which for the first time gives a bound that only depends on the control grid and not anymore on the number of the system’s controls. For the generalization of the control approximation problem with additional constraints, we derived a tailored branch-and-bound algorithm, which is based on the properties of the Lagrangian relaxation of the one-dimensional problem. This algorithm beats state-of-the-art commercial solvers for Mixed-Integer Linear Programs (MILPs) for this special approximation problem by several orders of magnitude. Overall, we present several, partially new modeling approaches for MIOCPs together with the accompanying structural properties. On this basis, we develop new theories for the approximation of certain relaxed solutions. We discuss the efficient implementation of the resulting structure exploiting algorithms. This leads to a deeper and better understanding of MIOCPs. We show the practicability of the theoretical observations with the help of four prototypical problems. The presented methods and algorithms allow on their basis the direct development of decision support and analysis tools in practice.

In this paper, we propose a distributed algorithm for scheduling divisible loads originating from single site on arbitrary networks. We first propose a mathematical model and formulate the scheduling problem as an optimization problem with an objective to minimize the processing time of the loads. A number of theoretical results on the solution of the optimization problem are derived. On the basis of these results, we propose our algorithm using the concept of virtual routing. The proposed algorithm has three attractive features - distributed working style, simple structure, in terms of implementation ease, and offers generalized approach for handling divisible load scheduling for any network topology. This is the first time in the divisible load scheduling literature that a distributed strategy is attempted.

Crow Search Algorithm is a novel Metaheuristic method based on the intelligent behavior of crows. It has been used to solve some optimization problems like engineering design problems, feature extraction and classification problems but has not been applied to vehicle routing problem. In this paper, crow search algorithm has been utilized to solve capacitated vehicle routing problem and the performance of it on various size-capacitated vehicle routing problem instances is analyzed. The factorial design ANOVA is used to determine the performance of crow search algorithm under different parameter settings on capacitated vehicle routing problem instances.

A fundamental problem in computer and communication networks is the wavelength assignment (WA) problem: given a set of routing paths on a network, assign wavelengths (channels) to the paths such that the paths with the same wavelength are edge-disjoint. The optimization problem here is to minimize the number of wavelengths. A popular network topology is a tree of rings. It is known NP-hard to find the minimum number of wavelengths for the WA problem on a tree of rings. Let L be the maximum number of paths on any edge in the network. Then L is a lower bound on the number of wavelengths for the WA problem. We give a polynomial time algorithm which uses at most 3L wavelengths for the WA problem on a tree of rings with node degree at most eight. This improves the previous result of 4L. We also show that some instances of the WA problem require at least 3L wavelengths on a tree of rings, implying that the 3L upper bound is optimal for the worst case instances. In addition, we prove that our algorithm has approximation ratios 2 and 2.5 for a tree of rings with node degrees at most four and six, respectively.

Fuzzy programming offers a powerful means of handling optimization problems with fuzzy parameters. Fuzzy programming has been used in different ways in the past. The particle swarm optimization (PSO) has been applied successfully to continuous nonlinear constrained optimization problems, neural network, etc. But we have not been found to use PSO for fuzzy programming in literature. In this paper, we combined with fuzzy simulation, neural network and PSO to produce a hybrid intelligent algorithm. Based on this hybrid intelligent algorithm, we introduced for solving fuzzy expected value models. Some numerical examples are given to illustrate the algorithm is effective and powerful.

With the increasing complexity of modern VLSI circuits, achieving high quality built-in self-test requires monitoring an increasingly large number of internal nodes. Due to the limitations in observing large numbers of nodes, it has become increasingly necessary to compact the output from a large number of lines to a small number of lines in a process known as space compaction. In the past, space compaction has usually been performed by circuit-independent compactors, such as the popular parity function. Recently, a class of circuit-specific space compactors, known as programmable space compactors (PSCs), has been proposed. A PSC is a highly effective space compactor tailored for a specific circuit under test (CUT) subjected to a specific set of test stimuli. Circuit-specific information such as the fault-free and expected faulty behaviour of a circuit can be used to choose compactors which have better fault coverage and/or lower area costs than the parity function. The drawback of PSCs is the difficulty involved in finding optimal PSCs for a circuit. The space of possible PSC functions is extremely large and grows exponentially with the number of lines to be compacted. Although searching for optimal PSC functions is a difficult problem, it is not impossible. This thesis presents a method for searching for combinational PSCs based upon genetic algorithms (GAs), a randomized search technique. Randomized search techniques, such as simulated annealing and genetic algorithms, have become increasingly popular in recent years as methods for solving search and optimization problems beyond the capabilities of classical techniques. This thesis develops a procedure for using circuit-specific information in the application of (GAs) to the search for optimal PSCs for a given CUT and a given set of test stimuli. A measure of the effectiveness, or fitness, of PSCs is developed as a means for guiding the genetic search. The factors used to assess the effectiveness of a PSC are its