Learning by on-line gradient descent

We study generalization in a committee machine with nonoverlapping receptive fields trained to implement a function of the same structure. Using the replica method, we calculate the generalization error in the limit of a large number of hidden units. For continuous weights the generalization error falls off asymptotically inversely proportional to α, the number of training examples per weight. For binary weights we find a discontinuous transition from poor to perfect generalization followed by a wide region of metastability. Broken replica symmetry is found within this region at low temperatures. The first-order transition occurs at a lower and the metastability limit at a higher value of α than in the simple perceptron.

An important problem in pattern recognition is that of pattern classification. The objective of classification is to determine a discriminant function which is consistent with the given training examples and performs reasonably well on an unlabeled test set of examples. The degree of performance of the classifier on the test examples, known as its generalization performance, is an important issue in the design of the classifier. It has been established that a good generalization performance can be achieved by providing the learner with a sufficiently large number of discriminative training examples. However, in many domains, it is infeasible or expensive to obtain a sufficiently large training set. Various mechanisms have been proposed in literature to combat this problem. Active Learning techniques (Angluin, 1998; Seung, Opper, & Sompolinsky, 1992) reduce the number of training examples required by carefully choosing discriminative training examples. Bootstrapping (Efron, 1979; Hamamoto, Uchimura & Tomita, 1997) and other pattern synthesis techniques generate a synthetic training set from the given training set. We present some of these techniques and propose some general mechanisms for pattern synthesis. Request access from your librarian to read this chapter's full text.

A central problem in learning is selection of an appropriate model. This is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate. In this article, we discuss the problem of model selection and generalization error estimation in the context of kernel regression models, e.g., kernel ridge regression, kernel subset regression or Gaussian process regression. Previously, a non-asymptotic generalization error estimator called the subspace information criterion (SIC) was proposed, that could be successfully applied to finite dimensional subspace models. SIC is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel Hilbert space (RKHS) H and the reproducing kernels centered on training sample points span the whole space H. These conditions hold only if dim H < l, where l < ∞ is the number of training examples. Therefore, SIC could be applied only to finite dimensional RKHSs. In this paper, we extend the range of applicability of SIC, and show that even if the reproducing kernels centered on training sample points do not span the whole space H, SIC is an unbiased estimator of an essential part of the generalization error. Our extension allows the use of any RKHSs including infinite dimensional ones, i.e., richer function classes commonly used in Gaussian processes, support vector machines or boosting. We further show that when the kernel matrix is invertible, SIC can be expressed in a much simpler form, making its computation highly efficient. In computer simulations on ridge parameter selection with real and artificial data sets, SIC is compared favorably with other standard model selection techniques for instance leave-one-out cross-validation or an empirical Bayesian method.

Modern classification techniques perform well when the number of training examples exceed the number of features. If, however, the number of features greatly exceed the number of training examples, then these same techniques can fail. To address this problem, we present a hierarchical Bayesian framework that shares information between features by modeling similarities between their parameters. We believe this approach is applicable to many sparse, high dimensional problems and especially relevant to those with both spatial and temporal components. One such problem is fMRI time series, and we present a case study that shows how we can successfully classify in this domain with 80,000 original features and only 2 training examples per class.

The study of the possibility of using an artificial neural network (ANN) to predict the hourly electricity consumption by the region of the Russian Federation was carried out. The data on electricity consumption and average daily temperature were selected as the initial data. The initial data were normalized so that they did not exceed the value of one. The number of training examples was 360 and each example had 3 inputs (the training dataset contained 1080 values). At the output, ANN calculated the hourly electricity consumption. A two-layer ANN was used for modeling. ANN was trained using the back propagation method and the conjugate gradient method. ANN training was carried out with an acceptable error of 0.00001 and was within 200 iterations of the conjugate gradient method. The use of an artificial neural network made it possible to satisfactorily describe the actual hourly electricity consumption. The average error was within 4%. The largest deviations of the actual and calculated values of hourly electricity consumption were observed around the 23rd, the 24th every day and amounted to 10–12%. The error increased as the forecast range increased. Calculations were carried out for a different number of neurons in the hidden layer: 7, 10, 13. These changes practically did not affect the forecast results. To increase the accuracy of the hourly forecast, it seems that the hourly temperature change during the day should be taken into account.

Phase unwrapping is one of the toughest problems in interferometric SAR processing. The main difficulties arise from the presence of point-like error sources, called residues, which occur mainly in close couples due to phase noise. We present an assessment of a local approach to the resolution of these problems by means of a neural network. Using a multi-layer perceptron, trained with the back- propagation scheme on a series of simulated phase images, fashion the best pairing strategies for close residue couples. Results show that god efficiencies and accuracies can have been obtained, provided a sufficient number of training examples are supplied. Results show that good efficiencies and accuracies can be obtained, provided a sufficient number of training examples are supplied. The technique is tested also on real SAR ERS-1/2 tandem interferometric images of the Matera test site, showing a good reduction of the residue density. The better results obtained by use of the neural network as far as local criteria are adopted appear justified given the probabilistic nature of the noise process on SAR interferometric phase fields and allows to outline a specifically tailored implementation of the neural network approach as a very fast pre-processing step intended to decrease the residue density and give sufficiently clean images to be processed further by more conventional techniques.

We propose a novel algorithm for the task of supervised discriminative distance learning by nonlinearly embedding vectors into a low dimensional Euclidean space. We work in the challenging setting where supervision is with constraints on similar and dissimilar pairs while training. The proposed method is derived by an approximate kernelization of a linear Mahalanobis-like distance metric learning algorithm and can also be seen as a kernel neural network. The number of model parameters and test time evaluation complexity of the proposed method are O(dD) where D is the dimensionality of the input features and d is the dimension of the projection space -- this is in contrast to the usual kernelization methods as, unlike them, the complexity does not scale linearly with the number of training examples. We propose a stochastic gradient based learning algorithm which makes the method scalable (w.r.t. the number of training examples), while being nonlinear. We train the method with up to half a million training pairs of 4096 dimensional CNN features. We give empirical comparisons with relevant baselines on seven challenging datasets for the task of low dimensional semantic category based image retrieval.

Most machine learning algorithms assume stationary environments, require a large number of training examples in advance, and begin the learning from scratch. In contrast, humans learn in changing environments with sequential training examples and leverage prior knowledge in new situations. To deal with real-world problems in changing environments, the ability to make human-like quick responses must be developed in machines. Many researchers have presented learning systems that assume the presence of hidden context and concept drift. In particular, several systems have been proposed that use ensembles of classifiers on sequential chunks of training examples. These systems can respond to gradual changes in large-scale data streams but have problems responding to sudden changes and leveraging prior knowledge of recurring contexts. Moreover, these are not pure online learning systems. We propose an online learning system that uses an ensemble of classifiers suited to recent training examples. We use experiments to show that this system can leverage prior knowledge of recurring contexts and is robust against various noise levels and types of drift.

On-the-fly learning systems are necessary for the deployment of general purpose robots. New training examples for such systems are often supplied by mentor interactions. Due to the cost of acquiring such examples, it is desirable to reduce the number of necessary interactions. Transfer learning has been shown to improve classification results for classes with small numbers of training examples by pooling knowledge from related classes. Standard practice in these works is to assume that the relationship between the transfer target and related classes is already known. In this work, we explore how previously learned categories, or related groupings of classes, can be used to transfer knowledge to novel classes without explicitly known relationships to them. We demonstrate an algorithm for determining the category membership of a novel class, focusing on the difficult case when few training examples are available. We show that classifiers trained via this method outperform classifiers optimized to learn the novel class individually when evaluated on both synthetic and real-world datasets.

We consider the use of “on-line” stopping rules to reduce the number of training examples needed to pat-learn. Rather than collect a large training sample that can be proved sufficient to eliminate all bad hypotheses a przorz, the idea is instead to observe training examples one-at-a-time and decide “on-line” whether to stop and return a hypothesis, or continue training. The primary benefit of this approach is that we can detect when a hypothesizer has actually ‘[converged,” and halt training before the standard fixed-sample-size bounds. This paper presents a series of such sequential learning procedures for: distribution-free pat-learning, “mist ake-bounded to pat” conversion, and distribution-specific pat-learning, respectively. We analyze the worst case expected training sample size of these procedures, and show that this is often smaller than existing fixed sample size bounds — while providing the exact same worst case pat-guarantees. We also provide lower bounds that show these reductions can at best involve constant (and possibly log) factors. However, empirical studies show that these sequential learning procedures actually use many times fewer training examples in practice.

In the last few years, object detection techniques have progressed immensely. Impressive detection results have been achieved for many objects such as faces [11, 14, 9] and cars [11]. The robustness of these systems emerges from a training stage utilizing thousands of positive examples. One approach to enable learning from a small set of training examples is to find an efficient set of features that accurately represent the target object. Unfortunately, automatically selecting such a feature set is a difficult task in itself. In this paper we present a novel feature selection method that is based on the notion of object categories. We assume that when learning to recognize a new object (like an apple) we also know a category it belongs to (fruit). We further assume that features that are useful for learning other objects in the same category (e.g. pear or orange) will also be useful for learning the novel object. This leads to a simple criterion for selecting features and building classifiers. We show that our method gives significant improvement in detection performance in challenging domains.

Multi-label learning deals with training examples each represented by a single instance while associated with multiple class labels. Due to the exponential number of possible label sets to be considered by the predictive model, it is commonly assumed that label correlations should be well exploited to design an effective multi-label learning approach. On the other hand, class-imbalance stands as an intrinsic property of multi-label data which significantly affects the generalization performance of the multi-label predictive model. For each class label, the number of training examples with positive labeling assignment is generally much less than those with negative labeling assignment. To deal with the class-imbalance issue for multi-label learning, a simple yet effective class-imbalance aware learning strategy called cross-coupling aggregation (COCOA) is proposed in this article. Specifically, COCOA works by leveraging the exploitation of label correlations as well as the exploration of class-imbalance simultaneously. For each class label, a number of multiclass imbalance learners are induced by randomly coupling with other labels, whose predictions on the unseen instance are aggregated to determine the corresponding labeling relevancy. Extensive experiments on 18 benchmark datasets clearly validate the effectiveness of COCOA against state-of-the-art multi-label learning approaches especially in terms of imbalance-specific evaluation metrics.

Combinatorial domains are of interest because they allow a large repertoire of behaviours to be described from a few relatively simple rules. However, the problem that combinatorial domains pose for machines that learn from examples, such as connectionist networks, is to learning the target behaviour adequately without a corresponding explosion in training examples. We show, both theoretically and empirically, that a feedforward network can generalize to an order of examples greater than that on which it was trained. Specifically, for the encoding of N-tuples, where the example space grows exponentially with N, only a polynomial number of training examples was required to achieve a fixed degree of accuracy over the entire domain.

Human activity recognition finds many applications in areas such as surveillance, and sports. Such a system classifies a spatio-temporal feature descriptor of a human figure in a video, based on training examples. However many classifiers face the constraints of the long training time, and the large size of the feature vector. Our method, due to the use of an Support Vector Machine (SVM) classifier, on an existing spatio-temporal feature descriptor resolves these problems in human activity recognition. Comparison of our system with existing classifiers using two standard datasets shows that our system is much superior in terms of the computational time, and either it surpasses or is on par with the existing recognition rates. It performs on par or marginally inferior to existing systems, when the number of training examples are a few due to the imbalance, although consistently better in terms of computation time.

We investigate a deep learning methodology that can produce more accurate behaviors for autonomous vehicles with a much smaller amount of training data than by using supervised learning alone. In this paper, we develop a Correction-Based Incremental Learning (CBIL) algorithm that adds additional training examples strategically selected from cases where the autonomous vehicle has made mistakes, and is repeated over multiple iterations to dramatically improve mean time to failure. CBIL can be thought of as an online mistake bound learning model that reduces the number of training examples needed to define robust decision boundaries, and is trained offline to solve the problem of catastrophic forgetting. We quantitatively benchmark the performance of CBIL using several experiments related to autonomous platooning performed in truck driving simulations and in the laboratory with mobile robots.