A General Framework for Decentralized Optimization With First-Order Methods

Tensor decomposition has been proved to be effective for solving many problems in signal processing and machine learning[1]. Recently, tensor decomposition finds its advantage for compressing deep neural networks. In many applications of deep neural networks, it is critical to reduce the number of parameters and computation workload to accelerate inference speed in deployment of the network. Modern deep neural network consists of multiple layers with multi-array weights where tensor decomposition is a natural way to perform compression. It is achieved by decomposing the weight tensors in convolutional layers or fully-connected layers with specified tensor ranks (e.g. canonical ranks, tensor train ranks). Conventional tensor decomposition in compressing deep neural networks selects the ranks manually that requires tedious human efforts to finetune the performance. To overcome this issue, we propose a novel rank selection scheme, which is inspired by reinforcement learning, to automatically select ranks in recently studied tensor ring decomposition in each convolutional layer. Experimental results validate that our learning based rank selection significantly outperforms hand-crafted rank selection heuristics on a number of benchmark datasets, for the purpose of effectively compressing deep neural networks while maintaining comparable accuracy.

The complex kernel adaptive filter (CKAF) has been widely applied to the complex-valued nonlinear problem in signal processing and machine learning. However, most of the CKAF applications involve the complex kernel least mean square (CKLMS) algorithms, which work in a pure complex or complexified reproducing kernel Hilbert space (RKHS). In this paper, we propose the generalized complex kernel affine projection (GCKAP) algorithms in the widely linear complex-valued RKHS (WL-RKHS). The proposed algorithms have two main notable features. One is that they provide a complete solution for both circular and non-circular complex nonlinear problems and show many performance improvements over the CKAP algorithms. The other is that the GCKAP algorithms inherit the simplicity of the CKLMS algorithm while reducing its gradient noise and boosting its convergence. The second-order statistical characteristics of WL-RKHS have also been developed. An augmented Gram matrix consists of a standard Gram matrix and a pseudo-Gram matrix. This decomposition provides more underlying information when the real and imaginary parts of the signal are correlated and learning is independent. In addition, some online sparsification criteria are compared comprehensively in the GCKAP algorithms, including the novelty criterion, the coherence criterion, and the angle criterion. Finally, two nonlinear channel equalization experiments with non-circular complex inputs are presented to illustrate the performance improvements of the proposed algorithms.

Low-rank matrix recovery is a fundamental problem in signal processing and machine learning. A recent very popular approach to recovering a low-rank matrix X is to factorize it as a product of two smaller matrices, i.e., X = UV <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</sup> , and then optimize over U, V instead of X. Despite the resulting non-convexity, recent results have shown that many factorized objective functions actually have benign global geometry-with no spurious local minima and satisfying the so-called strict saddle property-ensuring convergence to a global minimum for many local-search algorithms. Such results hold whenever the original objective function is restricted strongly convex and smooth. However, most of these results actually consider a modified cost function that includes a balancing regularizer. While useful for deriving theory, this balancing regularizer does not appear to be necessary in practice. In this work, we close this theory-practice gap by proving that the unaltered factorized non-convex problem, without the balancing regularizer, also has similar benign global geometry. Moreover, we also extend our theoretical results to the field of distributed optimization.

Feature learning that extracts concise and general- izable representations for data is one of the central problems in machine learning and signal processing. Sparse dictionary learning, also known as sparse coding, distinguishes from other feature learning techniques in sparsity exploitation, allowing the formulation of nonconvex optimizations that simultaneously uncover a structured dictionary and sparse representations. Despite the popularity of dictionary learning in applications, the landscapes of these optimizations that enable effective learning largely remain a mystery. This work characterizes the local optimization geometry for a simplified version of sparse coding where the L1 norm of the sparse coefficient matrix is minimized subject to orthogonal dictionary constraints. In particular, we show that the ground-truth dictionary and coefficient matrix are locally identifiable under the assumption that the coefficient matrix is sufficiently sparse and the number of training data columns is sufficiently large.

This dissertation presents novel learning and optimization frameworks for sparse sensor scheduling in large-scale complex networks. In the first part, efficient algorithms for sensor scheduling in dynamical networks are introduced, with guaranteed performance. In the second part, the applications of these frameworks are explored, focusing on experimental design for nonlinear biological models and visual feature selection for robotic navigation. The first framework addresses sensor scheduling in discrete-time, linear time-invariant systems, proposing a novel learning-based rounding method to transform weighted sensor schedules into sparse, unweighted schedules, while preserving similar observability performance. The second framework extends this approach to linear time-varying systems, employing a myopic strategy within a Kalman filtering framework to address non-submodular sensor scheduling in large-scale Discrete-Time Linear Time-Varying (DT-LTV) systems. A simple greedy algorithm is utilized, providing approximation bounds based on submodularity and curvature concepts. The third framework explores a non-myopic approach for DT-LTV systems through an online sparse sensor scheduling methodology, leveraging randomized algorithms to approximate fully-sensed systems with a constant average number of active sensors at each time step. Finally, a non-myopic approach within a Kalman filtering framework is considered for DT-LTV dynamics, incorporating both process and measurement noise. This approach utilizes recent advances in online learning and regret minimization to offer an adaptive sensor scheduling framework, effectively addressing submodular objectives. In the second part of this dissertation, the focus shifts to the application of these frameworks, specifically addressing the challenge of selecting an optimal subset of observations to improve parameter estimation, a common problem in machine learning and signal processing. For nonlinear measurement models, we go beyond standard linear Taylor approximations by adopting a quadratic approximation for greater accuracy in highly nonlinear cases. An efficient greedy selection method with theoretical utility bounds is proposed to optimally select experiments, specifically demonstrated using the Michaelis-Menten model in a simple enzyme reaction system. In the final section, a task-oriented framework is developed to improve visual-inertial navigation in robots, addressing constraints such as time and energy. The framework selects visual landmarks using a mean squared error based, non-submodular objective and a simplified dynamic model. To solve the NP-hard landmark selection problem, three efficient algorithms are proposed: a simple greedy method, a randomized variant for faster results, and a Taylor series-based approximation. The greedy and randomized variant approaches provide performance guarantees with respect to the optimal solution. Simulation results conducted for each proposed framework validate the effectiveness and superiority of these methods compared to existing techniques. Additionally, potential future research directions are discussed in the conclusion.--Author's abstract

As the scale and dimensionality of data continue to grow in many applications of data analytics (e.g., bioinformatics, finance, computer vision, medical informatics), it becomes critical to develop efficient and effective algorithms to solve numerous machine learning and data mining problems. This tutorial will focus on simple yet practically effective techniques and algorithms for big data analytics. In the first part, we plan to present the state-of-the-art large-scale optimization algorithms, including various stochastic gradient descent methods, stochastic coordinate descent methods and distributed optimization algorithms, for solving various machine learning problems. In the second part, we will focus on randomized approximation algorithms for learning from large-scale data. We will discuss i) randomized algorithms for low-rank matrix approximation; ii) approximation techniques for solving kernel learning problems; iii) randomized reduction methods for addressing the high-dimensional challenge. Along with the description of algorithms, we will also present some empirical results to facilitate understanding of different algorithms and comparison between them.

Markovian jumping systems have arisen naturally in the mathematical modeling of phenomena spanning disciplines in the social sciences, natural sciences, and engineering. This kind of stochastic dynamical systems can be employed to model the dynamics when parameters are subject to random abrupt changes due to sudden environment changes, subsystem switching, system noises, executor faults, and so forth. Much attention has been given to modeling, optimization, and real applications of such stochastic dynamical systems in the literature in recent years. As the advanced control and optimization will provide a basis for the design and application of such stochastic systems, these advanced techniques would result in substantial and sustainable benefits. The accepted papers in this special issue include stochastic stability, stabilization, stochastic control optimization, system modeling and identification methods, predictive control, signal processing, robust filtering, multiagent systems, networked control systems, time-delayed systems, neural networks, the Takagi-Sugeno fuzzy systems, simulated annealing, and fault detection methods.

Blind single-channel source separation is a long-standing machine learning and signal processing problem. Traditional blind source separation (BSS) algorithms were proposed to solve this task utilizing multiple signal constraints. Generative adversarial network (GAN) are free from statistical constraints and samples, but the role of adversarial training in the task of BSS has not been fully demonstrated. Therefore, a new separation network model that enables to learn the known separated signal distribution from the stepwise fine estimation of the unknown mixture distribution was presented in this paper, and a self-attention mechanism was introduced to solve the problem of the blurring details of the generated image by the generator which preserves image details in the process of image separation. Compared with the existing single-channel blind source separation algorithm based on generative adversarial network-neural egg separation (NES), the detailed information of this new separation algorithm is more prominent, and the source signal in the mixed image has been separated more effectively, and has better separation performance than the classic blind source separation algorithms.

This paper proposes and analyzes a communication-efficient distributed optimization framework for general nonconvex nonsmooth signal processing and machine learning problems under an asynchronous protocol. At each iteration, worker machines compute gradients of a known empirical loss function using their own local data, and a master machine solves a related minimization problem to update the current estimate. We prove that for nonconvex nonsmooth problems, the proposed algorithm converges with a sublinear rate over the number of communication rounds, coinciding with the best theoretical rate that can be achieved for this class of problems. Linear convergence is established without any statistical assumptions of the local data for problems characterized by composite loss functions whose smooth parts are strongly convex. Extensive numerical experiments verify that the performance of the proposed approach indeed improves -- sometimes significantly -- over other state-of-the-art algorithms in terms of total communication efficiency.

In this work, a gradient-based primal-dual method of multipliers is proposed for solving a class of linearly constrained non-convex problems. We show that with random initialization of the primal and dual variables, the algorithm is able to compute second-order stationary points (SOSPs) with probability one. Further, we present applications of the proposed method in popular signal processing and machine learning problems such as decentralized matrix factorization and decentralized training of overparameterized neural networks. One of the key steps in the analysis is to construct a new loss function for these problems such that the required convergence conditions (especially the gradient Lipschitz conditions) can be satisfied without changing the global optimal points.

Diffraction and holography are fertile areas for application of signal theory and processing. Recent work on 3DTV displays has posed particularly challenging signal processing problems. Various procedures to compute Rayleigh-Sommerfeld, Fresnel and Fraunhofer diffraction exist in the literature. Diffraction between parallel planes and tilted planes can be efficiently computed. Discretization and quantization of diffraction fields yield interesting theoretical and practical results, and allow efficient schemes compared to commonly used Nyquist sampling. The literature on computer-generated holography provides a good resource for holographic 3DTV related issues. Fast algorithms to compute Fourier, Walsh-Hadamard, fractional Fourier, linear canonical, Fresnel, and wavelet transforms, as well as optimization-based techniques such as best orthogonal basis, matching pursuit, basis pursuit etc., are especially relevant signal processing techniques for wave propagation, diffraction, holography, and related problems. Atomic decompositions, multiresolution techniques, Gabor functions, and Wigner distributions are among the signal processing techniques which have or may be applied to problems in optics. Research aimed at solving such problems at the intersection of wave optics and signal processing promises not only to facilitate the development of 3DTV systems, but also to contribute to fundamental advances in optics and signal processing theory.

Stochastic proximal gradient descent (Prox-SGD) is a standard optimization algorithm for solving stochastic composite optimization problems in machine learning. However, the existing convergence rate analysis of Prox-SGD is mostly focused on convergence in expectation. To this end, we provide an almost sure convergence rate analysis of Prox-SGD in two optimization settings: convexity and strong convexity, showing that any implementation of the stochastic algorithm will converge with probability one. Specifically, we show that the average-iterates almost sure convergence rate of the function values for Prox-SGD is arbitrarily close to the optimal rate o ( 1 / T ) in the convex setting. For strongly convex objective functions, we obtain the almost sure convergence rate of the iterative sequence, which is also arbitrarily close to the optimal rate o ( 1 / T ) . Last but not least, we establish the more challenging last-iterate convergence rate of the function values on convex problems, which contrasts with existing results on convergence for a weighted average.

Many problems in neural computation and statistical learning involve optimizations with nonnegativity constraints. In this article, we study convex problems in quadratic programming where the optimization is confined to an axis-aligned region in the nonnegative orthant. For these problems, we derive multiplicative updates that improve the value of the objective function at each iteration and converge monotonically to the global minimum. The updates have a simple closed form and do not involve any heuristics or free parameters that must be tuned to ensure convergence. Despite their simplicity, they differ strikingly in form from other multiplicative updates used in machine learning. We provide complete proofs of convergence for these updates and describe their application to problems in signal processing and pattern recognition.

Dictionary learning is a prototypical machine learning problem with broad applications throughout computer vision and signal processing. Given a set of training data, the goal is to learn a dictionary of prototype patterns that provide a sparse (parsimonious) representation of the data. Such a dictionary then enables a variety of problems to be solved by exploiting sparse structure; example problems include noise removal, super-resolution, deblurring, imputing missing data, and classification. Like many problems in machine learning, dictionary learning can be formulated as an optimization problem. Unfortunately, this optimization problem is non-convex, making it potentially difficult to solve to global optimality. Additional questions arise in how to deal with multi-modal data and distributed data, such as might be collected by multiple satellites, UAVs, etc. This presentation will summarize our team’s research on theory and algorithms for solving non-convex matrix-valued optimization problems such as dictionary learning, particularly in multi-modal, distributed, and streaming contexts. We welcome the opportunity to discuss further transitioning this technology for Naval applications.

Coreset, which is a summary of the original dataset in the form of a small weighted set in the same sample space, provides a promising approach to enable machine learning over distributed data. Although viewed as a proxy of the original dataset, each coreset is only designed to approximate the cost function of a specific machine learning problem, and thus different coresets are often required to solve different machine learning problems, increasing the communication overhead. We resolve this dilemma by developing robust coreset construction algorithms that can support a variety of machine learning problems. Motivated by empirical evidence that suitably-weighted k-clustering centers provide a robust coreset, we harden the observation by establishing theoretical conditions under which the coreset provides a guaranteed approximation for a broad range of machine learning problems, and developing both centralized and distributed algorithms to generate coresets satisfying the conditions. The robustness of the proposed algorithms is verified through extensive experiments on diverse datasets with respect to both supervised and unsupervised learning problems.