Multiplicative Updates for Nonnegative Quadratic Programming

Decentralized optimization to minimize a finite sum of functions, distributed over a network of nodes, has been a significant area within control and signal-processing research due to its natural relevance to optimal control and signal estimation problems. More recently, the emergence of sophisticated computing and large-scale data science needs have led to a resurgence of activity in this area. In this article, we discuss decentralized first-order gradient methods, which have found tremendous success in control, signal processing, and machine learning problems, where such methods, due to their simplicity, serve as the first method of choice for many complex inference and training tasks. In particular, we provide a general framework of decentralized first-order methods that is applicable to directed and undirected communication networks alike and show that much of the existing work on optimization and consensus can be related explicitly to this framework. We further extend the discussion to decentralized stochastic first-order methods that rely on stochastic gradients at each node and describe how local variance reduction schemes, previously shown to have promise in the centralized settings, are able to improve the performance of decentralized methods when combined with what is known as gradient tracking. We motivate and demonstrate the effectiveness of the corresponding methods in the context of machine learning and signal-processing problems that arise in decentralized environments.

Neural networks have been used over many years to help solve a large range of problems in pattern recognition, control and general signal processing. In this sense “neural computing” has generally been restricted to these types of domains. In contrast, here we investigate how neural networks can be adapted to actually solve numerical problems in algebra and geometry by modelling connections and transducer functions.

Pattern recognition is the study of automatic processing and interpretation of patterns using computers through mathematical techniques, where the environment and the objects are collectively referred to as “patterns”. It studies the problem of classifying samples by their attributes using computational methods. The main research directions of pattern recognition are computer vision, image processing, and natural language processing. In this paper, we discuss the application of a variant of reinforcement meta-learning, which combines experience replay and meta-learning in reinforcement learning, to the problem of continuous learning in pattern recognition. It also compares its accuracy with current solutions such as online learning for the continuous learning problem in pattern recognition. It mainly consists of reinforcement learning, reptilian meta-learning and experience replay in SGDM optimizer. It is finally concluded that for the problem of continuous learning in pattern recognition, its ability to preserve past learned knowledge with rapid learning of current knowledge is higher than the current online learning. Based on this, the effect of different parameters on the performance of this variant of reinforcement meta-learning is explored.

Class imbalance remains an open problem in pattern recognition, machine learning, and related fields. Many of the state-of-the-art classification algorithms tend to classify all unbalanced dataset patterns by assigning them to a majority class, thus failing to correctly classify a minority class. Associative memories are models used for pattern recall; however, they can also be employed for pattern classification. In this paper, a novel method for improving the classification performance of a hybrid associative classifier with translation (better known by its acronym in Spanish, CHAT) is presented. The extreme center points (ECP) method modifies the CHAT algorithm by exploring alternative vectors in a hyperspace for translating the training data, which is an inherent step of the original algorithm. We demonstrate the importance of our proposal by applying it to imbalanced datasets and comparing the performance to well-known classifiers by means of the balanced accuracy. The proposed method not only enhances the performance of the original CHAT algorithm, but it also outperforms state-of-the-art classifiers in four of the twelve analyzed datasets, making it a suitable algorithm for classification in imbalanced class scenarios.

As the volume of data grows at an unprecedented rate, large-scale data mining and knowledge discovery present a tremendous challenge. Rough set theory, which has been used successfully in solving problems in pattern recognition, machine learning, and data mining, centers around the idea that a set of distinct objects may be approximated via a lower and upper bound. In order to obtain the benefits that rough sets can provide for data mining and related tasks, efficient computation of these approximations is vital. The recently introduced cloud computing model, MapReduce, has gained a lot of attention from the scientific community for its applicability to large-scale data analysis. In previous research, we proposed a MapReduce-based method for computing approximations in parallel, which can efficiently process complete data but fails in the case of missing (incomplete) data. To address this shortcoming, three different parallel matrix-based methods are introduced to process large-scale, incomplete data. All of them are built on MapReduce and implemented on Twister that is a lightweight MapReduce runtime system. The proposed parallel methods are then experimentally shown to be efficient for processing large-scale data.

Artificial neural systems for computation were first proposed three quarters of a century ago and the concepts developed by the pioneers still shape the field today. The first generation of neural systems was developed in the nineteen forties in the context of analogue electronics and the theoretical research in logic and mathematics that led to the first digital computers in nineteen forties and fifties. The second generation of neural systems implemented on digital computers was born in the nineteen fifties and great progress was made in the subsequent half century with neural networks being applied to many problems in pattern recognition and machine learning. Through this history there has been an interplay between biologically inspired neural systems and their implementation by engineers on digital machines. This thesis concerns the third generation of neural networks, Spiking Neural Networks, which is making possible the creation of new kinds of brain inspired computing architectures that offer the potential to increase the level of realism and sophistication in terms of autonomous machine behaviour and cognitive computing. This thesis presents the development and demonstration of a new theoretical architecture for third generation neural systems, the Integrate-and-Fire based Spiking Neural Model with extended Neuro-modulated Spike Timing Dependent Plasticity capabilities. This proposed architecture overcomes the limitation of the homosynaptic architecture underlying existing implementations of spiking neural networks that it lacks a natural spike timing dependent plasticity regulation mechanism, and this results in ‘run away’ dynamics. To overcome this ad hoc procedures have been implemented to overcome the ‘run away’ dynamics that emerge from the use of spike timing dependent plasticity among other hebbian-based plasticity rules. The new heterosynaptic architecture presented, explicitly abstracts the modulation of complex biochemical mechanisms into a simplified mechanism that is suitable for the engineering of artificial systems with low computational complexity. Neurons work by receiving input signals from other neurons through synapses. The difference between homosynaptic and heterosynaptic plasticity is that, in the former the change in the properties of a synapse (e.g. synaptic efficacy) depends on the point to point activity in either of the sending and receiving neurons, in contrast for heterosynaptic plasticity the change in the properties of a synapse can be elicited by neurons that are not necessary presynaptic or postsynaptic to the synapse in question. The new architecture is tested by a number of implementations in simulated and real environments. This includes experiments with a simulation environment implemented in Netlogo, and an implementation using Lego Mindstorms as the physical robot platform. These experiments demonstrate the problems with the traditional Spike timing dependent plasticity homosynaptic architecture and how the new heterosynaptic approach can overcome them. It is concluded that the new theoretical architecture provides a natural, theoretically sound, and practical new direction for research into the role of modulatory neural systems applied to spiking neural networks.

From the Publisher: Pattern recognition has long been studied in relation to many different (and mainly unrelated) applications, such as remote sensing, computer vision, space research, and medical imaging. In this book Professor Ripley brings together two crucial ideas in pattern recognition; statistical methods and machine learning via neural networks. Unifying principles are brought to the fore, and the author gives an overview of the state of the subject. Many examples are included to illustrate real problems in pattern recognition and how to overcome them.This is a self-contained account, ideal both as an introduction for non-specialists readers, and also as a handbook for the more expert reader.

Machine learning and pattern recognition recently become a hot topic in computing world. This is due to the fast-growing of resources as well as techniques that make it easier to solve machine learning and pattern recognition problems. Problems that require machine learning solutions may be very simple for humans but actually can be very complex for machines to solve them. Face recognition is amongst those problems. Almost all human can easily recognize others without require specific knowledge to do it, different from machines which require its. This paper discussed face recognition task using machine learning strategies which involved Kernel Principal Component Analysis (KPCA) and Support Vector Machine (SVM) to identify person. KPCA extracted features from 2D image input and produced the important features of an image input. The extracted face features are recognized by SVM by classifying human face according to their stored identity in a database. SVM, which was basically a binary classifier worked by using one-against-one strategy to compare the face feature vector of a single test image to the stored face image in a face image database. Experiment results on grayscale images with size 92x112 pixels gave 96.25% of accuracy rate. Hence, KPCA and SVM for face recognition is a robust machine learning method.

Tensor decomposition has been proved to be effective for solving many problems in signal processing and machine learning[1]. Recently, tensor decomposition finds its advantage for compressing deep neural networks. In many applications of deep neural networks, it is critical to reduce the number of parameters and computation workload to accelerate inference speed in deployment of the network. Modern deep neural network consists of multiple layers with multi-array weights where tensor decomposition is a natural way to perform compression. It is achieved by decomposing the weight tensors in convolutional layers or fully-connected layers with specified tensor ranks (e.g. canonical ranks, tensor train ranks). Conventional tensor decomposition in compressing deep neural networks selects the ranks manually that requires tedious human efforts to finetune the performance. To overcome this issue, we propose a novel rank selection scheme, which is inspired by reinforcement learning, to automatically select ranks in recently studied tensor ring decomposition in each convolutional layer. Experimental results validate that our learning based rank selection significantly outperforms hand-crafted rank selection heuristics on a number of benchmark datasets, for the purpose of effectively compressing deep neural networks while maintaining comparable accuracy.

The complex kernel adaptive filter (CKAF) has been widely applied to the complex-valued nonlinear problem in signal processing and machine learning. However, most of the CKAF applications involve the complex kernel least mean square (CKLMS) algorithms, which work in a pure complex or complexified reproducing kernel Hilbert space (RKHS). In this paper, we propose the generalized complex kernel affine projection (GCKAP) algorithms in the widely linear complex-valued RKHS (WL-RKHS). The proposed algorithms have two main notable features. One is that they provide a complete solution for both circular and non-circular complex nonlinear problems and show many performance improvements over the CKAP algorithms. The other is that the GCKAP algorithms inherit the simplicity of the CKLMS algorithm while reducing its gradient noise and boosting its convergence. The second-order statistical characteristics of WL-RKHS have also been developed. An augmented Gram matrix consists of a standard Gram matrix and a pseudo-Gram matrix. This decomposition provides more underlying information when the real and imaginary parts of the signal are correlated and learning is independent. In addition, some online sparsification criteria are compared comprehensively in the GCKAP algorithms, including the novelty criterion, the coherence criterion, and the angle criterion. Finally, two nonlinear channel equalization experiments with non-circular complex inputs are presented to illustrate the performance improvements of the proposed algorithms.

Low-rank matrix recovery is a fundamental problem in signal processing and machine learning. A recent very popular approach to recovering a low-rank matrix X is to factorize it as a product of two smaller matrices, i.e., X = UV <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</sup> , and then optimize over U, V instead of X. Despite the resulting non-convexity, recent results have shown that many factorized objective functions actually have benign global geometry-with no spurious local minima and satisfying the so-called strict saddle property-ensuring convergence to a global minimum for many local-search algorithms. Such results hold whenever the original objective function is restricted strongly convex and smooth. However, most of these results actually consider a modified cost function that includes a balancing regularizer. While useful for deriving theory, this balancing regularizer does not appear to be necessary in practice. In this work, we close this theory-practice gap by proving that the unaltered factorized non-convex problem, without the balancing regularizer, also has similar benign global geometry. Moreover, we also extend our theoretical results to the field of distributed optimization.

We propose an efficient algorithm to solve the generalized Total Variation (TV) functional with a non-negativity constraint. This algorithm, which does not involve the solution of a linear system, but rather multiplicative updates only, can be used to solve the denoising and deconvolution problems. The derivation of our method is straightforward once the generalized TV functional is cast as a Non-negative Quadratic Programming (NQP) problem. The proposed algorithm offers a fair computational performance to solve the ℓ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -TV and ℓ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> -TV denoising and deconvolution problems and it is the fastest algorithm of which we are aware for general inverse problems involving a nontrivial forward linear operator and a non-negativity constraint.

The DSP3 Multiprocessor Project has produced a multi-GFLOP machine that is applicable to a variety of signal processing and pattern recognition problems. A companion program, the Ultra-Dense Project, is utilizing silicon-on-silicon multichip module (MCM) technology and innovative advanced packaging and physical design to achieve a parallel three-dimensional digital signal processor based on the DSP3. The authors present an overview and outline systems architecture the micro-interconnect technology (Si-on-Si MCM), and advances in physical design and packaging technology leading to an ultra-dense project demonstration. >

High-dimensional feature selection for genomic datasets

One of the central problems in machine learning and pattern recognition is how to deal with high-dimensional data either for visualization or for classification and clustering. Most of dimensionality reduction technologies, designed to cope with the curse of dimensionality, are based on Euclidean distance metric. In this work, we propose an unsupervised nonlinear dimensionality reduction method which attempt to preserve the distribution of input data, called distribution preserving embedding (DPE). It is done by minimizing the dissimilarity between the densities estimated in the original and embedded spaces. In theory, patterns in data can effectively be described by the distribution of the data. Therefore, DPE is able to discover the intrinsic pattern (structure) of data, including the global structures and the local structures. Additionally, DPE can be extended to cope with out-of-sample problem naturally. Extensive experiments on different data sets compared with other competing methods are reported to demonstrate the effectiveness of the proposed approach.