Beating the hold-out

An important problem in pattern recognition is that of pattern classification. The objective of classification is to determine a discriminant function which is consistent with the given training examples and performs reasonably well on an unlabeled test set of examples. The degree of performance of the classifier on the test examples, known as its generalization performance, is an important issue in the design of the classifier. It has been established that a good generalization performance can be achieved by providing the learner with a sufficiently large number of discriminative training examples. However, in many domains, it is infeasible or expensive to obtain a sufficiently large training set. Various mechanisms have been proposed in literature to combat this problem. Active Learning techniques (Angluin, 1998; Seung, Opper, & Sompolinsky, 1992) reduce the number of training examples required by carefully choosing discriminative training examples. Bootstrapping (Efron, 1979; Hamamoto, Uchimura & Tomita, 1997) and other pattern synthesis techniques generate a synthetic training set from the given training set. We present some of these techniques and propose some general mechanisms for pattern synthesis. Request access from your librarian to read this chapter's full text.

A central problem in learning is selection of an appropriate model. This is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate. In this article, we discuss the problem of model selection and generalization error estimation in the context of kernel regression models, e.g., kernel ridge regression, kernel subset regression or Gaussian process regression. Previously, a non-asymptotic generalization error estimator called the subspace information criterion (SIC) was proposed, that could be successfully applied to finite dimensional subspace models. SIC is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel Hilbert space (RKHS) H and the reproducing kernels centered on training sample points span the whole space H. These conditions hold only if dim H < l, where l < ∞ is the number of training examples. Therefore, SIC could be applied only to finite dimensional RKHSs. In this paper, we extend the range of applicability of SIC, and show that even if the reproducing kernels centered on training sample points do not span the whole space H, SIC is an unbiased estimator of an essential part of the generalization error. Our extension allows the use of any RKHSs including infinite dimensional ones, i.e., richer function classes commonly used in Gaussian processes, support vector machines or boosting. We further show that when the kernel matrix is invertible, SIC can be expressed in a much simpler form, making its computation highly efficient. In computer simulations on ridge parameter selection with real and artificial data sets, SIC is compared favorably with other standard model selection techniques for instance leave-one-out cross-validation or an empirical Bayesian method.

We propose a novel algorithm for the task of supervised discriminative distance learning by nonlinearly embedding vectors into a low dimensional Euclidean space. We work in the challenging setting where supervision is with constraints on similar and dissimilar pairs while training. The proposed method is derived by an approximate kernelization of a linear Mahalanobis-like distance metric learning algorithm and can also be seen as a kernel neural network. The number of model parameters and test time evaluation complexity of the proposed method are O(dD) where D is the dimensionality of the input features and d is the dimension of the projection space -- this is in contrast to the usual kernelization methods as, unlike them, the complexity does not scale linearly with the number of training examples. We propose a stochastic gradient based learning algorithm which makes the method scalable (w.r.t. the number of training examples), while being nonlinear. We train the method with up to half a million training pairs of 4096 dimensional CNN features. We give empirical comparisons with relevant baselines on seven challenging datasets for the task of low dimensional semantic category based image retrieval.

Evaluation of simple performance measures for tuning SVM hyperparameters

Average Case Analysis of Conjunctive Learning Algorithms

In this study, the face recognition task is applied on masked and unmasked faces using hand-crafted methods. Due to COVID-19 and masks, facial identification from unconstrained images became a hot topic. To avoid COVID-19, most people use masks outside. In many cases, typical facial recognition technology is useless. The majority of contemporary advanced face recognition methods are based on deep learning, which primarily relies on a huge number of training examples, however, masked face recognition may be investigated using hand-crafted approaches at a lower computing cost than using deep learning systems. A low-cost system is intended to be constructed for recognizing masked faces and compares its performance to that of face recognition systems that do not use masks. The proposed method fuses hand-crafted methods using feature-level fusion strategy. This study compares the performance of masked and unmasked face recognition systems. The experiments are undertaken on two publicly accessible datasets for masked face recognition: Masked Labeled Faces in the Wild (MLFW) and Cross-Age Labeled Faces in the Wild (CALFW). The best accuracy is achieved as 94.8% on MLFW dataset. The rest of the results on different train and test sets from CALFW and MLFW datasets are encouraging compared to the state-of-the-art models.

Modern classification techniques perform well when the number of training examples exceed the number of features. If, however, the number of features greatly exceed the number of training examples, then these same techniques can fail. To address this problem, we present a hierarchical Bayesian framework that shares information between features by modeling similarities between their parameters. We believe this approach is applicable to many sparse, high dimensional problems and especially relevant to those with both spatial and temporal components. One such problem is fMRI time series, and we present a case study that shows how we can successfully classify in this domain with 80,000 original features and only 2 training examples per class.

This study aimed to investigate a deep learning model to classify amyloid plaque deposition in the brain PET images of patients suspected of Alzheimer's disease. A retrospective study was conducted on patients who were suspected of having a mild cognitive impairment or dementia, and brain amyloid 18 F florapronol PET/computed tomography images were obtained from 2019 to 2022. Brain PET images were visually assessed by two nuclear medicine specialists, who classified them as either positive or negative. Image rotation was applied for data augmentation. The dataset was split into training and testing sets at a ratio of 8 : 2. For the convolutional neural network (CNN) analysis, stratified k-fold ( k = 5) cross-validation was applied using training set. Trained model was evaluated using testing set. A total of 175 patients were included in this study. The average age at the time of PET imaging was 70.4 ± 9.3 years and included 77 men and 98 women (44.0% and 56.0%, respectively). The visual assessment revealed positivity in 62 patients (35.4%) and negativity in 113 patients (64.6%). After stratified k-fold cross-validation, the CNN model showed an average accuracy of 0.917 ± 0.027. The model exhibited an accuracy of 0.914 and an area under the curve of 0.958 in the testing set. These findings affirm the model's high reliability in distinguishing between positive and negative cases. The study verifies the potential of the CNN model to classify amyloid positive and negative cases using brain PET images. This model may serve as a supplementary tool to enhance the accuracy of clinical diagnoses.

We investigate a deep learning methodology that can produce more accurate behaviors for autonomous vehicles with a much smaller amount of training data than by using supervised learning alone. In this paper, we develop a Correction-Based Incremental Learning (CBIL) algorithm that adds additional training examples strategically selected from cases where the autonomous vehicle has made mistakes, and is repeated over multiple iterations to dramatically improve mean time to failure. CBIL can be thought of as an online mistake bound learning model that reduces the number of training examples needed to define robust decision boundaries, and is trained offline to solve the problem of catastrophic forgetting. We quantitatively benchmark the performance of CBIL using several experiments related to autonomous platooning performed in truck driving simulations and in the laboratory with mobile robots.

An incremental one-class learning algorithm is proposed for the purpose of outlier detection. Outliers are identified by estimating - and thresholding - the probability distribution of the training data. In the early stages of training a non-parametric estimate of the training data distribution is obtained using kernel density estimation. Once the number of training examples reaches the maximum computationally feasible limit for kernel density estimation, we treat the kernel density estimate as a maximally-complex Gaussian mixture model, and keep the model complexity constant by merging a pair of components for each new kernel added. This method is shown to outperform a current state-of-the-art incremental one-class learning algorithm (Incremental SVDD [5]) on a variety of datasets, while requiring only an upper limit on model complexity to be specified.KeywordsTraining DataGaussian Mixture ModelOutlier DetectionKernel Density EstimationNormal ClassThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Machine learning algorithms can be expensive to deploy, in particular, those used in robotics applications that perform many variations of the same task. Solutions to one variation of a task may be found via Reinforcement Learning algorithms, and are typically modeled as a vector of N parameters encoding the robot's behavior policy. When N is large or executing robot trials is time-consuming, searching in the space of solutions becomes prohibitively expensive. In this paper, we introduce a method that allows robots to generalize behaviors by analyzing solutions to a small number of previously-trained related tasks. This allows for approximate policies for novel tasks to be rapidly estimated. We present a method that achieves this type of generalization by performing nonlinear regression directly on the policy manifold — i.e., the solution space spanned as we change the parameters describing tasks. Because tasks are typically described by few parameters, the corresponding policy manifold has few degrees of freedom, which leads to low-dimensional surfaces. We exploit this property to construct a function that maps task parameters to policy parameters (a parameterized skill). Our method uses manifold clustering techniques to deal with discontinuous manifolds, a challenging situation arising from physical obstacles or robot constraints. We evaluate our method on a set of robot manipulation tasks and show that it can efficiently estimate policies for novel tasks from a small number of training examples.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Most machine learning algorithms assume stationary environments, require a large number of training examples in advance, and begin the learning from scratch. In contrast, humans learn in changing environments with sequential training examples and leverage prior knowledge in new situations. To deal with real-world problems in changing environments, the ability to make human-like quick responses must be developed in machines. Many researchers have presented learning systems that assume the presence of hidden context and concept drift. In particular, several systems have been proposed that use ensembles of classifiers on sequential chunks of training examples. These systems can respond to gradual changes in large-scale data streams but have problems responding to sudden changes and leveraging prior knowledge of recurring contexts. Moreover, these are not pure online learning systems. We propose an online learning system that uses an ensemble of classifiers suited to recent training examples. We use experiments to show that this system can leverage prior knowledge of recurring contexts and is robust against various noise levels and types of drift.

Phase unwrapping is one of the toughest problems in interferometric SAR processing. The main difficulties arise from the presence of point-like error sources, called residues, which occur mainly in close couples due to phase noise. We present an assessment of a local approach to the resolution of these problems by means of a neural network. Using a multi-layer perceptron, trained with the back- propagation scheme on a series of simulated phase images, fashion the best pairing strategies for close residue couples. Results show that god efficiencies and accuracies can have been obtained, provided a sufficient number of training examples are supplied. Results show that good efficiencies and accuracies can be obtained, provided a sufficient number of training examples are supplied. The technique is tested also on real SAR ERS-1/2 tandem interferometric images of the Matera test site, showing a good reduction of the residue density. The better results obtained by use of the neural network as far as local criteria are adopted appear justified given the probabilistic nature of the noise process on SAR interferometric phase fields and allows to outline a specifically tailored implementation of the neural network approach as a very fast pre-processing step intended to decrease the residue density and give sufficiently clean images to be processed further by more conventional techniques.

Application of contrast-enhanced CT-driven multimodal machine learning models for pulmonary metastasis prediction in head and neck adenoid cystic carcinoma.