Ho–Kashyap classifier with early stopping for regularization

Early stopping is an extremely common tool to minimize overfitting, which would otherwise be a cause of poor generalization of the model to novel data. However, early stopping is a heuristic that, while effective, primarily relies on ad hoc parameters and metrics. Optimizing when to stop remains a challenge. In this paper, we suggest that for some biomedical applications, a natural dichotomy of invasive/non-invasive measurements, or more generally proximal vs distal measurements of a biological system can be exploited to provide objective advice on early stopping. We discuss the conditions where invasive measurements of a biological process should provide better predictions than non-invasive measurements, or at best offer parity. Hence, if data from an invasive measurement are available locally, or from the literature, that information can be leveraged to know with high certainty whether a model of non-invasive data is overfitted. We present paired invasive/non-invasive cardiac and coronary artery measurements from two mouse strains, one of which spontaneously develops type 2 diabetes, posed as a classification problem. Examination of the various stopping rules shows that generalization is reduced with more training epochs and commonly applied stopping rules give widely different generalization error estimates. The use of an empirically derived training ceiling is demonstrated to be helpful as added information to leverage early stopping in order to reduce overfitting.

Early stopping (ES) methods reduce the average number of iterations required for decoding turbo codes. These methods monitor the generated outputs along the iterations to terminate the loop once this output reaches a satisfactory level of reliability. It is the norm among the published ES methods to be applied after the output of the second decoder becomes available. In other words, an iteration index includes both decoders. In this paper, we propose a more efficient way to apply an ES method. This improvement relies on the fact that almost any ES check can be executed twice per iteration, i.e., after each decoder instead of after only the second one. In all the cases, the used ES algorithm considers the two most recent outputs from the constituent decoders. Results show further reduction in the average iterations for the decoding, which is practically important for conserving decoding resources.

При решении практически значимых задач глобальной оптимизации целевая функция зачастую имеет высокую размерность и вычислительную сложность, а также нетривиальный ландшафт. Исследования показывают, что зачастую одного метода оптимизации недостаточно для эффективного решения такого рода задач – необходима гибридизация нескольких методов оптимизации.

Matrix-pattern-oriented Ho–Kashyap classifier with regularization learning

A central problem in learning is selection of an appropriate model. This is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate. In this article, we discuss the problem of model selection and generalization error estimation in the context of kernel regression models, e.g., kernel ridge regression, kernel subset regression or Gaussian process regression. Previously, a non-asymptotic generalization error estimator called the subspace information criterion (SIC) was proposed, that could be successfully applied to finite dimensional subspace models. SIC is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel Hilbert space (RKHS) H and the reproducing kernels centered on training sample points span the whole space H. These conditions hold only if dim H < l, where l < ∞ is the number of training examples. Therefore, SIC could be applied only to finite dimensional RKHSs. In this paper, we extend the range of applicability of SIC, and show that even if the reproducing kernels centered on training sample points do not span the whole space H, SIC is an unbiased estimator of an essential part of the generalization error. Our extension allows the use of any RKHSs including infinite dimensional ones, i.e., richer function classes commonly used in Gaussian processes, support vector machines or boosting. We further show that when the kernel matrix is invertible, SIC can be expressed in a much simpler form, making its computation highly efficient. In computer simulations on ridge parameter selection with real and artificial data sets, SIC is compared favorably with other standard model selection techniques for instance leave-one-out cross-validation or an empirical Bayesian method.

The Sixth International Symposium on Neural Networks (ISNN 2009)

Over-fitting is a ubiquitous problem in machine learning, and a variety of techniques to avoid over-fitting the training sample have proven highly effective, including early stopping, regularization, and ensemble methods. However, while over-fitting in training is widely appreciated and its avoidance now a standard element of best practice, over-fitting can also occur in model selection. This form of over-fitting can significantly degrade generalization performance, but has thus far received little attention. For example the kernel and regularization parameters of a support vector machine are often tuned by optimizing a cross-validation based model selection criterion. However the cross-validation estimate of generalization performance will inevitably have a finite variance, such that its minimizer depends on the particular sample on which it is evaluated, and this will generally differ from the minimizer of the true generalization error. Therefore if the cross-validation error is aggressively minimized, generalization performance may be substantially degraded. In general, the smaller the amount of data available, the higher the variance of the model selection criterion, and hence the more likely over-fitting in model selection will be a significant problem. Similarly, the more hyper-parameters to be tuned in model selection, the more easily the variance of the model selection criterion can be exploited, which again increases the likelihood of over-fitting in model selection. Over-fitting in model selection is empirically demonstrated to pose a substantial pitfall in the application of kernel learning methods and Gaussian process classifiers. Furthermore, evaluation of machine learning methods can easily be significantly biased unless the evaluation protocol properly accounts for this type of over-fitting. Fortunately the common solutions to avoiding over-fitting in training also appear to be effective in avoiding over-fitting in model selection. Three examples are presented based on regularization of the model selection criterion, early stopping in model selection and minimizing the number of hyper-parameters to be tuned during model selection.

We consider estimators obtained by iterates of the conjugate gradient (CG) algorithm applied to the normal equation of prototypical statistical inverse problems. Stopping the CG algorithm early induces regularisation, and optimal convergence rates of prediction and reconstruction error are established in wide generality for an ideal oracle stopping time. Based on this insight, a fully data-driven early stopping rule τis constructed, which also attains optimal rates, provided the error in estimating the noise level is not dominant. The error analysis of CG under statistical noise is subtle due to its nonlinear dependence on the observations. We provide an explicit error decomposition and identify two terms in the prediction error, which share important properties of classical bias and variance terms. Together with a continuous interpolation between CG iterates, this paves the way for a comprehensive error analysis of early stopping. In particular, a general oracle-type inequality is proved for the prediction error at τ. For bounding the reconstruction error, a more refined probabilistic analysis, based on concentration of self-normalised Gaussian processes, is developed. The methodology also provides some new insights into early stopping for CG in deterministic inverse problems. A numerical study for standard examples shows good results in practice for early stopping at τ.

A limited amount of training data is a well-known challenge in the application of deep learning methods. This issue is particularly relevant in biomedical signal processing, such as the analysis of electrocardiograms, due to the labor-intensive nature of data preparation, which requires the involvement of qualified specialists. Self-supervised learning methods, originally developed in such domains as natural language processing and computer vision, have emerged as a potential approach to addressing this challenge and are increasingly being explored in biomedical signal processing. However, direct adaptation of self-supervised learning techniques from other domains does not fully account for ECG-specific characteristics, such as quasi-periodicity, localized morphological features, and susceptibility to noise. This highlights the relevance of developing ECG-specific self-supervised learning methods. This study presents a novel self-supervised learning approach for pretraining neural networks on unlabeled ECG data. The proposed method is based on predicting the short consecutive signal segment using a preceding one and a learned representation vector. The representation extraction and prediction models are trained jointly on the MIMIC-ECG-IV dataset using backpropagation to minimize the mean squared error between the predicted and original signal segments. As an example of a downstream task, a linear binary classifier was trained on the PTB-XL dataset to diagnose pathological conditions using Lead I. The number of training examples for each diagnosis was limited to thirty-four samples. Firstly, the representation model was pre-trained on the unlabeled MIMIC-ECG-IV dataset, and then linear classifiers were trained on the learned representations for each selected diagnosis in PTB-XL. A comparison was also conducted with a randomly initialized representation model trained jointly with the classifier in a fully supervised manner. The proposed method was evaluated against adaptations of Contrastive Learning, Contrastive Predictive Coding, and Masked Autoencoders method. To ensure a controlled experimental setup, implementations of all considered methods were developed using a unified codebase and shared architectural components. Experimental results demonstrated a significant advantage of all self-supervised learning approaches over joint training of feature extraction and classification models. The proposed SSL method outperformed other tested approaches, particularly for diagnoses with subtle short-term morphological features, such as atrial fibrillation and flutter. These findings suggest the potential for further research in developing ECG-specific self-supervised learning methods as a promising approach to improving neural network performance in scenarios with limited labeled data.

Globalized and localized matrix-pattern-oriented classification machine

An implicit algorithm of solving Navier–Stokes equations to simulate flows in anisotropic porous media

Evaluation of simple performance measures for tuning SVM hyperparameters

The empirical error on a test set, the hold-out estimate, often is a more reliable estimate of generalization error than the observed error on the training set, the training estimate. K-fold cross validation is used in practice with the hope of being more accurate than the hold-out estimate without reducing the number of training examples. We argue that the k-fold estimate does in fact achieve this goal. Specifically, we show that for any nontrivial learning problem and learning algorithm that is insensitive to example ordering, the k-fold estimate is strictly more accurate than a single hold-out estimate on 1/k of the data, for 2 < k < n (k = n is leave-one-out), based on its variance and all higher moments. Previous bounds were termed sanitycheck because they compared the k-fold estimate to the training estimate and, further, restricted the VC dimension and required a notion of hypothesis stability [2]. In order to avoid these dependencies, we consider a k-fold hypothesis that is a randomized combination or average of the k individual hypotheses. We introduce progressive validationas another possible improvement on the hold-out estimate. This estimate of the generalization error is, in many ways, as good as that of a single hold-out, but it uses an average of half as many examples for testing. The procedure also involves a hold-out set, but after an example has been tested, it is added to the training set and the learning algorithm is rerun.

Dictionary learning is an unsupervised learning method to abstract image data into a set of learned basis vectors. In prior work, the efficacy of the K-SVD dictionary learning algorithm in suppressing reverberation in volumetric photoacoustic microscopy (PAM) data was demonstrated. In this work, we compare the K-SVD algorithm against the method of optimal directions (MOD). The generalization error and reverberation suppression performance of the two algorithms were compared. The K-SVD was found to have a lower average generalization error (5.69x104 ±9.09x103 (a.u.)) when compared to the MOD (8.27x104 ±1.33x104 (a.u.)) for identical training data, initialization, sparsity (3 atoms per A-line) and number of iterations (5). Both algorithms were observed to suppress the reverberation to a similar extent (18.8 ± 1.12 dB for the K-SVD and 18.3 ± 1.2 dB for the MOD). Our data show that irrespective of the method used, sparse dictionary learning can significantly suppress reverberations in PAM.

Data mining applications explore large amounts of heterogeneous data in search of consistent information. In such a challenging context, empirical learning methods aim to optimize prediction on unseen data, and an accurate estimate of the generalization error is of paramount importance. The paper shows that the theoretical formulation based on the Vapnik-Chervonenkis dimension (d vc ) can be of practical interest when applied to clustering methods for data-mining applications. The presented research adopts the K-Winner Machine (KWM) as a clustering-based, semi-supervised classifier; in addition to fruitful theoretical properties, the model provides a general criterion for evaluating the applicability of Vapnik's generalization predictions in data mining. The general approach is verified experimentally in the practical problem of detecting intrusions in computer networks. Empirical results prove that the KWM model can effectively support such a difficult classification task and combine unsupervised and supervised.