Fast Training of Effective Multi-class Boosting Using Coordinate Descent Optimization

When all training instances and labels are considered all together in a single optimization problem, multi-label support and core vector machines (i.e., Rank-SVM and Rank-CVM) are formulated as quadratic programming (QP) problems with equality and bounded constraints, whose training procedures have a sub-linear convergence rate. Therefore it is highly desirable to design and implement a novel efficient SVM-type multi-label algorithm. In this paper, through applying pairwise constraints between relevant and irrelevant labels, and defining an approximate ranking loss, we generalize binary Lagrangian support vector machine (LSVM) to construct its multi-label form (Rank-LSVM), resulting into a strictly convex QP problem with non-negative constraints only. Particularly, each training instance is associated with a block of variables and all variables are divided naturally into manageable blocks. Consequently we build an efficient training procedure for Rank-LSVM using random block coordinate descent method with a linear convergence rate. Moreover a heuristic strategy is applied to reduce the number of support vectors. Experimental results on twelve data sets demonstrate that our method works better according to five performance measures, and averagely runs 15 and 107 times faster and has 9 and 15% fewer support vectors, compared with Rank-CVM and Rank-SVM.

Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradient method that randomly updates block-coordinate variables by using block composite subgradient mapping. This method converges asymptotically to critical points with proven sublinear convergence rate for certain optimality measures. Second, we develop a randomly permuted CD method with two alternating steps: linearizing the concave part and cycling through variables. We prove asymptotic convergence to critical points and sublinear complexity rate for objectives with both smooth and concave parts. Third, we extend accelerated coordinate descent (ACD) to nonsmooth and nonconvex optimization to develop a novel randomized proximal DC algorithm whereby we solve the subproblem inexactly by ACD. Convergence is guaranteed with at most a few number of ACD iterations for each DC subproblem, and convergence complexity is established for identification of some approximate critical points. Fourth, we further develop the third method to minimize certain ill-conditioned nonconvex functions: weakly convex functions with high Lipschitz constant to negative curvature ratios. We show that, under specific criteria, the ACD-based randomized method has superior complexity compared to conventional gradient methods. Finally, an empirical study on sparsity-inducing learning models demonstrates that CD methods are superior to gradient-based methods for certain large-scale problems.

We present a novel randomized block coordinate descent method for the minimization of a convex composite objective function. The method uses (approximate) partial second-order (curvature) information, so that the algorithm performance is more robust when applied to highly nonseparable or ill conditioned problems. We call the method Flexible Coordinate Descent (FCD). At each iteration of FCD, a block of coordinates is sampled randomly, a quadratic model is formed about that block and the model is minimized approximately/inexactly to determine the search direction. An inexpensive line search is then employed to ensure a monotonic decrease in the objective function and acceptance of large step sizes. We present several high probability iteration complexity results to show that convergence of FCD is guaranteed theoretically. Finally, we present numerical results on large-scale problems to demonstrate the practical performance of the method.

Most optimization methods for logistic regression or maximum entropy solve the primal problem. They range from iterative scaling, coordinate descent, quasi-Newton, and truncated Newton. Less efforts have been made to solve the dual problem. In contrast, for linear support vector machines (SVM), methods have been shown to be very effective for solving the dual problem. In this paper, we apply coordinate descent methods to solve the dual form of logistic regression and maximum entropy. Interestingly, many details are different from the situation in linear SVM. We carefully study the theoretical convergence as well as numerical issues. The proposed method is shown to be faster than most state of the art methods for training logistic regression and maximum entropy.

With the rapid development of information science and statistics, significant improvement in the ability to handle huge-sized data has been seen, including linear regression. However, traditional statistical methods remain as a problem when the number of samples is less than the rapidly increasing data dimensions and constructing a mathematical model in high-dimensional scenario may work it out. Hopefully linear regression becomes kind of solvable problem as far as singular matrix is transformed into nonsingular matrix through introduction of L2 norm as penalty term. Nevertheless, this method has to be ungraded by a new method titled ‘Lasso’ when options of variables is inaccessible to it. Coordinate descent method, alternating direction method of multipliers and proximal gradient descent method are gateways to Lasso, among others. The coordinate descent method gets the minimum value of each component on basis of L1 penalty term. Alternating direction method of multipliers the minimum value by processing decomposed objective functions separately. The proximal gradient descent method is a method of Taylor expansion at a point where initial value is selected. This paper has showed the results of simulation experiments on the three algorithms in terms of accuracy and convergence speed. It reveals that alternating direction method of multipliers, superior to the rest two both by speed and accuracy, is followed by The proximal gradient descent method, while The coordinate descent method is not equal.

The paper aims at finding an effective language for human-computer interaction in the context of the introduction of Virtual Reality and Artificial Intelligence within the framework of modern semiotics. A variant of the sign problem statement and an attempt to apply an analogy with thermodynamic complex systems for its solution are presented. Based on the ontological model of the semiotic spiral of cognition and activity, a vector representation of the semiosis cycle in the orthogonal space of coordinates of the semantic, syntactic and pragmatic dimensions is proposed. Movement along the spiral of cognition and activity forms epistemological oppositions of understanding and expression, which are codified in knowledge of a receptive and projective nature. Understanding describes the transition from a sign to its meaning, that is, the solution of the decoding problem, and expression describes the transition from the meaning to the sign denoting it, that is, the solution of the coding problem. Based on this representation, the problem of effective semiosis is formulated as a process of alternating the stages of understanding and expression, clarifying the system of using signs. To solve this problem, an analogy with a thermodynamic system can be used, in which the stages of transformation and exchange of energy described by macroscopic quantities also alternate. To perform optimization, a coordinate descent method is proposed. It is possible to move from an object to a sign by selecting the best semantic match, simplifying the sign in the syntactic sense and selecting a pragmatic rule. Conversely, it is proposed to select a certain reference sign and simplify the pragmatic rule, syntactic representation or semantic match. The ideal cycle of semiosis provides for a return to the same object that was used when selecting the sign in the absence of loss of meaning. In a real cycle, an iterative spiral coordinate descent is performed, during which an iterative refinement of the sign-object pair is performed. The implementation of the proposed approach in practice allows increasing the efficiency of user interfaces of hardware and software complexes of Virtual Reality with high immersion, as well as developing recommendations for the implementation of prompt engineering in Artificial Intelligence systems.

The Nonnegative Least Squares (NNLS) formulation arises in many important regression problems. We present a novel coordinate descent method which differs from previous approaches in that we do not explicitly maintain complete gradient information. Empirical evidence shows that our approach outperforms a state-of-the-art NNLS solver in computation time for calculating radiation dosage for cancer treatment problems.

GLMNET proposed by Friedman et al. is an algorithm for generalized linear models with elastic net. It has been widely applied to solve L1-regularized logistic regression. However, recent experiments indicated that the existing GLMNET implementation may not be stable for large-scale problems. In this paper, we propose an improved GLMNET to address some theoretical and implementation issues. In particular, as a Newton-type method, GLMNET achieves fast local convergence, but may fail to quickly obtain a useful solution. By a careful design to adjust the effort for each iteration, our method is efficient regardless of loosely or strictly solving the optimization problem. Experiments demonstrate that the improved GLMNET is more efficient than a state-of-the-art coordinate descent method.

As the flatness control system is a multi-variable control system, the key issue for high-precision flatness is the determination of the optimal adjustments of flatness actuators. In order to determine these the first step is the establishment of a multi-variable control model in the flatness control process, by which the actual flatness control problem has been reduced to a non-linear optimisation problem with box-constraints. Around this optimisation problem, characteristics and applicability of current optimisation algorithms were analysed. Based on the coordinate descent method, a new multi-variable optimisation algorithm with global convergence was proposed. In the algorithm, the problem was transformed into a series of univariate optimisations which could be solved by sequentially searching along coordinate directions. In order to make the objective function decrease rapidly, and ensure that each iteration are carried out within the feasible region, the accelerating step method is adopted to determine the iterative step size. A series of feasible points have been obtained through a series of iterations, which make the objective function decrease gradually until the optimal solution has been obtained. Finally, the algorithm has been tested by numerical experiments with production data of actual flatness control process, and applied to the 1450 mm tandem cold mill. Numerical experiments and application show that the proposed algorithm not only can satisfy the requirements of response speed, but also have a good accuracy, which can provide a reference for the realisation of high-precision flatness control processes of cold rolled strip.

The paper is devoted to the convergence properties of finite-difference local descent algorithms in global optimization problems with a special γ-convex structure. It is assumed that the objective function can be closely approximated by some smooth convex function. Stability properties of the perturbed gradient descent and coordinate descent methods are investigated. Basing on this results some global optimization properties of finite-difference local descent algorithms, in particular, coordinate descent method, are discovered. These properties are not inherent in methods using exact gradients.

Energy control plays an important role for underwater sensor networks. On one hand, due to the limitation of energy and bandwidth, quantization needs to be introduced while tracking; on the other hand, static sensor nodes have limited computing capability. In contrast, autonomous underwater vehicle (AUV) has high mobility and computing capability which could perform tracking energy control very well. Therefore, a tracking energy control algorithm assisted by AUV is proposed in this paper. First, transmission bits and AUV movement trajectory are simultaneously considered to minimize the energy consumption of sensor nodes under the constraint of PCRLB. Then, facing mixed-integer multivariate optimization problem above, relax and block coordinate descent method are used to interactively find optimal solution. Simulation results show that, compared with other algorithms, the proposed algorithm can achieve the minimum energy consumption under the given constraints.KeywordsUnderwater sensor networkAutonomous underwater vehicleTracking energy controlBlock coordinate descent method

Research on the fairness of spectral clustering has gradually increased attention. Normally, existing methods of fair spectral clustering add a fairness constraint to the original objective function so that fairness is guaranteed. However, similar to the solver of traditional spectral clustering, that of fairness spectral clustering has to relax a discrete value condition into an arbitrary one, which leads to the deterioration of both fairness and clustering quality. Moreover, the eigen-problem is inevitable in the solver, which takes O(n3) time complexity and is not available for large-scale data. In this paper, we propose a fair spectral clustering algorithm by employing the coordinate descent method to find the solution. As the relaxation of the discreteness condition is discarded, the fairness is improved. Furthermore, we refine the process of coordinate descent by avoiding redundant calculations, and as a result, the time complexity is reduced from O(n3) to O(n2). Additionally, the importance of clustering quality and fairness is symmetric; hence, we achieve a trade-off between them by adjusting the parameters. The experimental findings, obtained from both real-world and synthetic datasets, clearly illustrate that our proposal delivers superior fairness and clustering quality with the best BAL compared to other fair clustering methods. In addition, our method is more efficient than existing fair spectral clustering algorithms.

Block coordinate descent (BCD) methods approach optimization problems by performing gradient steps along alternating subgroups of coordinates. This is in contrast to full gradient descent, where a gradient step updates all coordinates simultaneously. BCD has been demonstrated to accelerate the gradient method in many practical large-scale applications. Despite its success no convergence analysis for inverse problems is known so far. In this paper, we investigate the BCD method for solving linear inverse problems. As the main theoretical result, we show that for operators having a particular tensor product form, the BCD method combined with an appropriate stopping criterion yields a convergent regularization method. To illustrate the theory, we perform numerical experiments comparing the BCD and the full gradient descent method for a system of integral equations. We also present numerical tests for a nonlinear inverse problem not covered by our theory, namely one-step inversion in multispectral X-ray tomography.

This article investigates the resource allocation of a reconfigurable intelligent surface (RIS)-aided joint communication and sensing (JCAS) system in a coal mine scenario. In the JCAS system, an RIS is implemented at the corner of the zigzag tunnels to improve the complicated wireless environment, where ground obstacles frequently block direct links. In addition, a wireless backhaul base station with a limited energy budget is deployed in the depth of the mine to sense the target area and provide Internet of Things (IoT) services and communication services for users. Furthermore, a data center is placed on the ground to analyze the obtained data and route the communication data. Under this deployment, a joint optimization problem of RIS phase-shift matrix, RIS element switches, and area sensing time is proposed. We aim to maximize the successful sensed bits under total completion time, and maximum transmit power constraints. In order to solve this problem, an iterative algorithm is proposed. The successive convex approximation (SCA)-based algorithm is used for the RIS phase-shift matrix optimization subproblem. For the sensing time optimization subproblem, the quadratic approximation method is proposed to optimize the number of area perceptions. The coordinate descent method is utilized to optimize the RIS element switches. Simulation results show that the energy efficiency is improved by up to 38%, and 7% increases the specific data size compared with the benchmark solutions.

In this paper, we propose a non-negative feature selection/feature grouping (nnFSG) method for general sign-constrained high-dimensional regression problems that allows regression coefficients to be disjointly homogeneous, with sparsity as a special case. To solve the resulting non-convex optimization problem, we provide an algorithm that incorporates the difference of convex programming, augmented Lagrange and coordinate descent methods. Furthermore, we show that the aforementioned nnFSG method recovers the oracle estimate consistently, and that the mean-squared errors are bounded. Additionally, we examine the performance of our method using finite sample simulations and applying it to a real protein mass spectrum dataset.