Plane Based Optimization Method: A Method for Better Global Minimum Cost Function Localization

In this chapter, we focus on introducing a distributed online optimization problem for a set of nodes communicating on a time-varying unbalanced directed network, while considering the problem of how to preserve the privacy of their local cost functions. The main goal of this set of nodes is to cooperatively minimize the sum of all locally known convex cost functions (global cost function). We propose a differentially private distributed stochastic subgradient-push algorithm, named DP-DSSP, to solve such an optimization problem in a collaborative and distributed manner. The algorithm ensures that nodes interact with their in-neighbors and collectively optimize the global cost function. Unlike most existing distributed algorithms that do not consider privacy issues, DP-DSSP successfully protects the privacy of participating nodes through a differential privacy strategy, which is more practical in applications involving sensitive information, such as military affairs or medical treatment. An important feature of DP-DSSP is that it handles distributed online optimization problems in the context of time-varying unbalanced directed networks. Theoretical analysis shows that DP-DSSP can effectively protect differential privacy and can achieve sublinear regret. The tradeoff between the level of privacy and accuracy of DP-DSSP is also revealed. Moreover, DP-DSSP is able to handle arbitrarily large but uniformly bounded delays in the communication link. Finally, simulation experiments confirm the usefulness of DP-DSSP and the findings of this chapter.KeywordsDistributed online optimizationDifferential privacyStochastic subgradient-push algorithmTime-varying directed networksCommunication delays

In this paper, we analyse the performance of the closed-loop Whiplash gradient descent algorithm [1] for L-smooth convex cost functions. Using numerical experiments, we study the algorithm’s performance for convex cost functions, for different condition numbers. We analyse the convergence of the momentum sequence using symplectic integration and introduce the concept of relaxation sequences which analyses the non-classical character of the whiplash method. Under the additional assumption of invexity, we establish a momentum-driven adaptive convergence rate. Furthermore, we introduce an energy method for predicting the convergence rate with convex cost functions for closed-loop inertial gradient dynamics, using an integral anchored energy function and a novel lower bound asymptotic notation, by exploiting the bounded nature of the solutions. Using this, we establish a polynomial convergence rate for the whiplash inertial gradient system, for a family of scalar quadratic cost functions and an exponential rate for a quadratic scalar cost function.

Nowadays, the influence of machine learning algorithms on our lives is booming gradually, and gradient descent (GD) method is a typical big data analysis method that can be applied to various machine learning algorithms. Traditional GD method has its advantages in dealing with the optimal parameter finding issue for partial loss functions, but the optimal parameter finding process may be trapped into local minimum sometimes if the loss function with certain convexity; moreover, traditional GD cannot meet the requirement of high accuracy. Fractional order GD, which is the core of fixed memory step GD, has better performance by comparing with traditional GD, but still have some problems in the aspect of convergence. On the other hand, t-distributed machine learning algorithm for dimensionality reduction, can reduce the dimensionality of data in high-dimensional space to two-dimensional or three-dimensional space, and can convert the original data into more comprehensible images. In order to solve the problem mentioned above reasonably, in this paper, a fixed-memory step GD method combined with t-distributed stochastic neighbor embedding (t-SNE) is employed to replace the traditional GD method, so as to keep the consistency of low-dimensional data being mapped from high-dimensional data, and to achieve efficient data visualization. The effectiveness of proposed t-SNE algorithm based on fixed memory step gradient descent method has been verified finally via numerical experiment.

Existing blind adaptive equalizers that use nonconvex cost functions and stochastic gradient descent suffer from lack of global convergence to an equalizer setup that removes sufficient ISI when an FIR equalizer is used. The authors impose convexity on the cost function and anchoring of the equalizer away from the all-zero setup. They establish that there exists a globally convergent blind equalization strategy for 1D pulse amplitude modulation (PAM) systems with bounded input data (discrete or continuous) even when the equalizer is truncated. The resulting cost function is a constrained l/sub 1/ norm of the joint impulse response of the channel and the equalizer. The results apply to arbitrary linear channels (provided there are no unit circle zeros) and apply regardless of the initial ISI (that is whether the eye is initially open or closed). They also show a globally convergent stochastic gradient scheme based on an implementable approximation of the l/sub 1/ cost function. >

We explore the striking mathematical connections that exist between market scoring rules, cost function based prediction markets, and no-regret learning. We show that any cost function based prediction market can be interpreted as an algorithm for the commonly studied problem of learning from expert advice by equating trades made in the market with losses observed by the learning algorithm. If the loss of the market organizer is bounded, this bound can be used to derive an O(sqrt(T)) regret bound for the corresponding learning algorithm. We then show that the class of markets with convex cost functions exactly corresponds to the class of Follow the Regularized Leader learning algorithms, with the choice of a cost function in the market corresponding to the choice of a regularizer in the learning problem. Finally, we show an equivalence between market scoring rules and prediction markets with convex cost functions. This implies that market scoring rules can also be interpreted naturally as Follow the Regularized Leader algorithms, and may be of independent interest. These connections provide new insight into how it is that commonly studied markets, such as the Logarithmic Market Scoring Rule, can aggregate opinions into accurate estimates of the likelihood of future events.

This paper revisits how private cross-ownership affects cost-reducing R&D, production quantities, consumer surplus, firms’ profits and social welfare in a mixed oligopoly with a convex cost function. We show that increasing cross-ownership reduces private firms’ R&D but increases the semi-public firm’s R&D. Further, both consumer surplus and social welfare could be enhanced through cross-ownership when the convexity of cost function is small. However, increasing cross-ownership could hurt consumers and social welfare when the convexity of cost function is large. Our finding highlights that the convexity of cost function is very important to the welfare implications of cross-ownership. Hence, antitrust authorities should be more careful when assessing the welfare effects of partial ownership in different industries.

The successful implementation of downhill search engines in radiotherapy optimization algorithms depends on the absence of local minima in the search space. Such techniques are much faster than stochastic optimization methods but may become trapped in local minima if they exist. A technique known as ‘configuration space analysis’ was applied to examine the search space of cost functions used in radiotherapy beam-weight optimization algorithms. A downhill-simplex beam-weight optimization algorithm was run repeatedly to produce a frequency distribution of final cost values. By plotting the frequency distribution as a function of final cost, the existence of local minima can be determined. Common cost functions such as the quadratic deviation of dose to the planning target volume (PTV), integral dose to organs-at-risk (OARs), dose-threshold and dose-volume constraints for OARs were studied. Combinations of the cost functions were also considered. The simple cost function terms such as the quadratic PTV dose and integral dose to OAR cost function terms are not susceptible to local minima. In contrast, dose-threshold and dose-volume OAR constraint cost function terms are able to produce local minima in the example case studied.

Artificial neural network (ANN) has been commonly used to deal with many problems. However, since this algorithm applies backpropagation algorithms based on gradient descent (GD) technique to look for the best solution, the network may face major risks of being entrapped in local minima. To overcome those drawbacks of ANN, in this work, we propose a novel ANN working parallel with metaheuristic algorithms (MAs) to train the network. The core idea is that first, (1) GD is applied to increase the convergence speed. (2) If the network is stuck in local minima, the capacity of the global search technique of MAs is employed. (3) After escaping from local minima, the GD technique is applied again. This process is applied until the target is achieved. Additionally, to increase the efficiency of the global search capacity, a hybrid of particle swarm optimization and genetic algorithm (PSOGA) is employed. The effectiveness of ANNPSOGA is assessed using both numerical models and measurement. The results demonstrate that ANNPSOGA provides higher accuracy than traditional ANN, PSO, and other hybrid ANNs (even a higher level of noise is employed) and also considerably decreases calculational cost compared with PSO.

We present an approach for the optimization of the live wire algorithm applied to 3D medical images. Our method restricts the computation of the cost function to relevant areas and considers regionally specific properties of the object boundary. As a consequence, precise contours can be obtained in reduced computation and interaction time. For the calculation of the cost function on the current image slice, the nearest contour on an adjacent slice is taken as reference. The reference contour is divided into local segments and the image pixels are classified into regions with respect to their distance to the contour segments. The size of these regions is controlled by a given maximum distance. Cost function parameters are learned separately from every local contour segment of the reference slice and define the cost function for the respective region on the current slice. We used the local cost computation for the interactive definition of object contours, as well as for the optimization of interpolated contours between user-defined contours. Applied to CT and MR data of the liver, our method showed considerable advantages over the conventional algorithm based on a global cost function, particularly for objects with inhomogeneities or with different surrounding tissue.

An optimization method of adjoint assimilation based on the cumulative gradient: An application for correction of the bottom friction coefficient in the tidal wave model

Stochastic optimization with randomized smoothing for image registration.

Most blind QAM equalization algorithms base their parameter adaptation on the minimization of non-convex cost functions. These cost functions may exhibit local minima which can cause undesirable convergence to equalizer parameter settings, resulting in insufficient removal of the intersymbol interference In contrast, this work considers the minimization of a special convex cost function of the equalizer output in combination with an arbitrary non-degenerate linear constraint. We establish the property that the cost minimization in the space of the equalizer parameters is achieved on a compact set that contains at least one ideal zero-ISI equalizer parameter setting. Generically this compact set consists of a single point reflecting an ideal global convergence result, implying zero intersymbol interference. Further, through general abstract methods we demonstrate that this desirable convergence property is largely independent of the input constellation geometry and independent of the linear constraint chosen. The methods are developed for and motivated by the important classes of QAM constellations used in practice.

The Asia Pacific Information and Communication Technology Alliance (APICTA) Awards has been held for 12 years, rewarding the most innovative solutions in different categories of Information and Communication Technology (ICT). To maintain professionalism, judges are nominated from each economy, and appointed to panels of different categories. Judge assignment is a difficult task, since it has to optimize between expertise, distribution of workloads, fairness and sometimes even political correctness. In this paper, we describe our experience in analyzing and automating the APICTA judge assignment process using Soft Constraint Programming for the 13 th APICTA hosted in Hong Kong on November, 2013. We chose the weighted constraint satisfaction (WCSP) framework since both hard constraints and preferences can be modeled by cost functions. Consistency algorithms can effect strong propagation by redistributing costs among cost functions. We observe that a number of restrictions in the judge assignment problem involves counting. In our first attempt, we utilized the Soft_Among var global cost function for these counting conditions but we could not solve the problem within a day. Soft_GCC val is another possible global cost function to model counting, which is what we used in the second attempt. We can compute the optimum in a few hours, which is far from practical.We apply similar techniques as Régin to show that the combination of Soft_GCC val and Soft_Among var is flow-based. We further prove that the combination results in a flow-based projection-safe cost function, meaning that soft arc consistencies can be enforced efficiently. By using this combination in our final model, we can solve the judge assignment problem within a few minutes. We consider this a success story where theory and practice meet.

Blind adaptive channel equalizers are important devices to remove channel distortion in high data-rate, bandlimited digital communication systems when the transmission of a training sequence is impractical or very costly. Traditional blind equalization algorithms adapt the equalizer parameters to minimize some specially designed non-MSE cost functions. These algorithms can experience local convergence problems and can thereby result in insufficient or no removal of channel distortion. We present a new quadrature amplitude modulated blind equalization scheme that is globally convergent in the equalizer parameter space to a compact set containing the desired ideal equalizer parameter setting. Our new algorithm is based on a convex cost function and a linear constraint on the equalizer parameters. For a generic class of channels, this new algorithm results in the equalizer parameter convergence to a unique global minimum achieving intersymbol interference suppression and carrier phase error removal. Different implementation approaches are assessed and simulation results are shown to confirm the theoretical global convergence of the new algorithm.

In the context of neural network procedures, it is proved that gradient descent on a surface defined by a sum of squared errors can fail to separate families of vectors. Each output is assumed to be a differentiable monotone transformation (typically the logistic) of a linear combination of inputs. Several examples are given of two families of vectors for which a linear combination exists that will serve to separate the two families. However, the minimum cost solution does not yield the desired combination. The examples include several cases where there are no local minima, as well as a one-layer system showing local minima with a large basin of attraction. In contrast to the perceptron convergence theorem, which proves that the perceptron architecture, there is no convergence theorem for gradient descent which would allow correct classification. The theorem disproves the presumption made in recent years, that barring local minima, gradient descent will find the best set of weights for a given problem. >