An improved GLMNET for l1-regularized logistic regression

With the rapid development of information science and statistics, significant improvement in the ability to handle huge-sized data has been seen, including linear regression. However, traditional statistical methods remain as a problem when the number of samples is less than the rapidly increasing data dimensions and constructing a mathematical model in high-dimensional scenario may work it out. Hopefully linear regression becomes kind of solvable problem as far as singular matrix is transformed into nonsingular matrix through introduction of L2 norm as penalty term. Nevertheless, this method has to be ungraded by a new method titled ‘Lasso’ when options of variables is inaccessible to it. Coordinate descent method, alternating direction method of multipliers and proximal gradient descent method are gateways to Lasso, among others. The coordinate descent method gets the minimum value of each component on basis of L1 penalty term. Alternating direction method of multipliers the minimum value by processing decomposed objective functions separately. The proximal gradient descent method is a method of Taylor expansion at a point where initial value is selected. This paper has showed the results of simulation experiments on the three algorithms in terms of accuracy and convergence speed. It reveals that alternating direction method of multipliers, superior to the rest two both by speed and accuracy, is followed by The proximal gradient descent method, while The coordinate descent method is not equal.

We present a novel column generation based boosting method for multi-class classification. Our multi-class boosting is formulated in a single optimization problem as in [1]. Different from most existing multi-class boosting methods, which use the same set of weak learners for all the classes, we train class specified weak learners (i.e., each class has a different set of weak learners). We show that using separate weak learner sets for each class leads to fast convergence, without introducing additional computational overhead in the training procedure. To further make the training more efficient and scalable, we also propose a fast coordinate descent method for solving the optimization problem at each boosting iteration. The proposed coordinate descent method is conceptually simple and easy to implement in that it is a closed-form solution for each coordinate update. Experimental results on a variety of datasets show that, compared to a range of existing multi-class boosting methods, the proposed method has much faster convergence rate and better generalization performance in most cases. We also empirically show that the proposed fast coordinate descent algorithm needs less training time than the MultiBoost algorithm in [1].KeywordsTraining TimeColumn GenerationMaster ProblemWeak LearnerCoordinate DescentThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradient method that randomly updates block-coordinate variables by using block composite subgradient mapping. This method converges asymptotically to critical points with proven sublinear convergence rate for certain optimality measures. Second, we develop a randomly permuted CD method with two alternating steps: linearizing the concave part and cycling through variables. We prove asymptotic convergence to critical points and sublinear complexity rate for objectives with both smooth and concave parts. Third, we extend accelerated coordinate descent (ACD) to nonsmooth and nonconvex optimization to develop a novel randomized proximal DC algorithm whereby we solve the subproblem inexactly by ACD. Convergence is guaranteed with at most a few number of ACD iterations for each DC subproblem, and convergence complexity is established for identification of some approximate critical points. Fourth, we further develop the third method to minimize certain ill-conditioned nonconvex functions: weakly convex functions with high Lipschitz constant to negative curvature ratios. We show that, under specific criteria, the ACD-based randomized method has superior complexity compared to conventional gradient methods. Finally, an empirical study on sparsity-inducing learning models demonstrates that CD methods are superior to gradient-based methods for certain large-scale problems.

The paper is devoted to the convergence properties of finite-difference local descent algorithms in global optimization problems with a special γ-convex structure. It is assumed that the objective function can be closely approximated by some smooth convex function. Stability properties of the perturbed gradient descent and coordinate descent methods are investigated. Basing on this results some global optimization properties of finite-difference local descent algorithms, in particular, coordinate descent method, are discovered. These properties are not inherent in methods using exact gradients.

We present a novel randomized block coordinate descent method for the minimization of a convex composite objective function. The method uses (approximate) partial second-order (curvature) information, so that the algorithm performance is more robust when applied to highly nonseparable or ill conditioned problems. We call the method Flexible Coordinate Descent (FCD). At each iteration of FCD, a block of coordinates is sampled randomly, a quadratic model is formed about that block and the model is minimized approximately/inexactly to determine the search direction. An inexpensive line search is then employed to ensure a monotonic decrease in the objective function and acceptance of large step sizes. We present several high probability iteration complexity results to show that convergence of FCD is guaranteed theoretically. Finally, we present numerical results on large-scale problems to demonstrate the practical performance of the method.

Most optimization methods for logistic regression or maximum entropy solve the primal problem. They range from iterative scaling, coordinate descent, quasi-Newton, and truncated Newton. Less efforts have been made to solve the dual problem. In contrast, for linear support vector machines (SVM), methods have been shown to be very effective for solving the dual problem. In this paper, we apply coordinate descent methods to solve the dual form of logistic regression and maximum entropy. Interestingly, many details are different from the situation in linear SVM. We carefully study the theoretical convergence as well as numerical issues. The proposed method is shown to be faster than most state of the art methods for training logistic regression and maximum entropy.

As the flatness control system is a multi-variable control system, the key issue for high-precision flatness is the determination of the optimal adjustments of flatness actuators. In order to determine these the first step is the establishment of a multi-variable control model in the flatness control process, by which the actual flatness control problem has been reduced to a non-linear optimisation problem with box-constraints. Around this optimisation problem, characteristics and applicability of current optimisation algorithms were analysed. Based on the coordinate descent method, a new multi-variable optimisation algorithm with global convergence was proposed. In the algorithm, the problem was transformed into a series of univariate optimisations which could be solved by sequentially searching along coordinate directions. In order to make the objective function decrease rapidly, and ensure that each iteration are carried out within the feasible region, the accelerating step method is adopted to determine the iterative step size. A series of feasible points have been obtained through a series of iterations, which make the objective function decrease gradually until the optimal solution has been obtained. Finally, the algorithm has been tested by numerical experiments with production data of actual flatness control process, and applied to the 1450 mm tandem cold mill. Numerical experiments and application show that the proposed algorithm not only can satisfy the requirements of response speed, but also have a good accuracy, which can provide a reference for the realisation of high-precision flatness control processes of cold rolled strip.

On a method of coordinate descent

Energy control plays an important role for underwater sensor networks. On one hand, due to the limitation of energy and bandwidth, quantization needs to be introduced while tracking; on the other hand, static sensor nodes have limited computing capability. In contrast, autonomous underwater vehicle (AUV) has high mobility and computing capability which could perform tracking energy control very well. Therefore, a tracking energy control algorithm assisted by AUV is proposed in this paper. First, transmission bits and AUV movement trajectory are simultaneously considered to minimize the energy consumption of sensor nodes under the constraint of PCRLB. Then, facing mixed-integer multivariate optimization problem above, relax and block coordinate descent method are used to interactively find optimal solution. Simulation results show that, compared with other algorithms, the proposed algorithm can achieve the minimum energy consumption under the given constraints.KeywordsUnderwater sensor networkAutonomous underwater vehicleTracking energy controlBlock coordinate descent method

Block coordinate descent (BCD) methods approach optimization problems by performing gradient steps along alternating subgroups of coordinates. This is in contrast to full gradient descent, where a gradient step updates all coordinates simultaneously. BCD has been demonstrated to accelerate the gradient method in many practical large-scale applications. Despite its success no convergence analysis for inverse problems is known so far. In this paper, we investigate the BCD method for solving linear inverse problems. As the main theoretical result, we show that for operators having a particular tensor product form, the BCD method combined with an appropriate stopping criterion yields a convergent regularization method. To illustrate the theory, we perform numerical experiments comparing the BCD and the full gradient descent method for a system of integral equations. We also present numerical tests for a nonlinear inverse problem not covered by our theory, namely one-step inversion in multispectral X-ray tomography.

The paper aims at finding an effective language for human-computer interaction in the context of the introduction of Virtual Reality and Artificial Intelligence within the framework of modern semiotics. A variant of the sign problem statement and an attempt to apply an analogy with thermodynamic complex systems for its solution are presented. Based on the ontological model of the semiotic spiral of cognition and activity, a vector representation of the semiosis cycle in the orthogonal space of coordinates of the semantic, syntactic and pragmatic dimensions is proposed. Movement along the spiral of cognition and activity forms epistemological oppositions of understanding and expression, which are codified in knowledge of a receptive and projective nature. Understanding describes the transition from a sign to its meaning, that is, the solution of the decoding problem, and expression describes the transition from the meaning to the sign denoting it, that is, the solution of the coding problem. Based on this representation, the problem of effective semiosis is formulated as a process of alternating the stages of understanding and expression, clarifying the system of using signs. To solve this problem, an analogy with a thermodynamic system can be used, in which the stages of transformation and exchange of energy described by macroscopic quantities also alternate. To perform optimization, a coordinate descent method is proposed. It is possible to move from an object to a sign by selecting the best semantic match, simplifying the sign in the syntactic sense and selecting a pragmatic rule. Conversely, it is proposed to select a certain reference sign and simplify the pragmatic rule, syntactic representation or semantic match. The ideal cycle of semiosis provides for a return to the same object that was used when selecting the sign in the absence of loss of meaning. In a real cycle, an iterative spiral coordinate descent is performed, during which an iterative refinement of the sign-object pair is performed. The implementation of the proposed approach in practice allows increasing the efficiency of user interfaces of hardware and software complexes of Virtual Reality with high immersion, as well as developing recommendations for the implementation of prompt engineering in Artificial Intelligence systems.

The Nonnegative Least Squares (NNLS) formulation arises in many important regression problems. We present a novel coordinate descent method which differs from previous approaches in that we do not explicitly maintain complete gradient information. Empirical evidence shows that our approach outperforms a state-of-the-art NNLS solver in computation time for calculating radiation dosage for cancer treatment problems.

This article investigates the resource allocation of a reconfigurable intelligent surface (RIS)-aided joint communication and sensing (JCAS) system in a coal mine scenario. In the JCAS system, an RIS is implemented at the corner of the zigzag tunnels to improve the complicated wireless environment, where ground obstacles frequently block direct links. In addition, a wireless backhaul base station with a limited energy budget is deployed in the depth of the mine to sense the target area and provide Internet of Things (IoT) services and communication services for users. Furthermore, a data center is placed on the ground to analyze the obtained data and route the communication data. Under this deployment, a joint optimization problem of RIS phase-shift matrix, RIS element switches, and area sensing time is proposed. We aim to maximize the successful sensed bits under total completion time, and maximum transmit power constraints. In order to solve this problem, an iterative algorithm is proposed. The successive convex approximation (SCA)-based algorithm is used for the RIS phase-shift matrix optimization subproblem. For the sensing time optimization subproblem, the quadratic approximation method is proposed to optimize the number of area perceptions. The coordinate descent method is utilized to optimize the RIS element switches. Simulation results show that the energy efficiency is improved by up to 38%, and 7% increases the specific data size compared with the benchmark solutions.

In this paper, we propose a non-negative feature selection/feature grouping (nnFSG) method for general sign-constrained high-dimensional regression problems that allows regression coefficients to be disjointly homogeneous, with sparsity as a special case. To solve the resulting non-convex optimization problem, we provide an algorithm that incorporates the difference of convex programming, augmented Lagrange and coordinate descent methods. Furthermore, we show that the aforementioned nnFSG method recovers the oracle estimate consistently, and that the mean-squared errors are bounded. Additionally, we examine the performance of our method using finite sample simulations and applying it to a real protein mass spectrum dataset.

Parameter estimation from multiple measurement vectors (MMVs) is a fundamental problem in many signal processing applications, e.g., spectral analysis and direction-of- arrival estimation. Recently, this problem has been address using prior information in form of a jointly sparse signal structure. A prominent approach for exploiting joint sparsity considers mixed-norm minimization in which, however, the problem size grows with the number of measurements and the desired resolution, respectively. In this work we derive an equivalent, compact reformulation of the $\ell_{2,1}$ mixed-norm minimization problem which provides new insights on the relation between different existing approaches for jointly sparse signal reconstruction. The reformulation builds upon a compact parameterization, which models the row-norms of the sparse signal representation as parameters of interest, resulting in a significant reduction of the MMV problem size. Given the sparse vector of row-norms, the jointly sparse signal can be computed from the MMVs in closed form. For the special case of uniform linear sampling, we present an extension of the compact formulation for gridless parameter estimation by means of semidefinite programming. Furthermore, we derive in this case from our compact problem formulation the exact equivalence between the $\ell_{2,1}$ mixed-norm minimization and the atomic-norm minimization. Additionally, for the case of irregular sampling or a large number of samples, we present a low complexity, grid-based implementation based on the coordinate descent method.