Early Stopping Based on Repeated Significance

Early stopping (ES) methods reduce the average number of iterations required for decoding turbo codes. These methods monitor the generated outputs along the iterations to terminate the loop once this output reaches a satisfactory level of reliability. It is the norm among the published ES methods to be applied after the output of the second decoder becomes available. In other words, an iteration index includes both decoders. In this paper, we propose a more efficient way to apply an ES method. This improvement relies on the fact that almost any ES check can be executed twice per iteration, i.e., after each decoder instead of after only the second one. In all the cases, the used ES algorithm considers the two most recent outputs from the constituent decoders. Results show further reduction in the average iterations for the decoding, which is practically important for conserving decoding resources.

Boosting is one of the most significant advances in machine learning for classification and regression. In its original and computationally flexible version, boosting seeks to minimize empirically a loss function in a greedy fashion. The resulting estimator takes an additive function form and is built iteratively by applying a base estimator (or learner) to updated samples depending on the previous iterations. An unusual regularization technique, early stopping, is employed based on CV or a test set. This paper studies numerical convergence, consistency and statistical rates of convergence of boosting with early stopping, when it is carried out over the linear span of a family of basis functions. For general loss functions, we prove the convergence of boosting’s greedy optimization to the infinimum of the loss function over the linear span. Using the numerical convergence result, we find early-stopping strategies under which boosting is shown to be consistent based on i.i.d. samples, and we obtain bounds on the rates of convergence for boosting estimators. Simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting. As a side product, these results also reveal the importance of restricting the greedy search step-sizes, as known in practice through the work of Friedman and others. Moreover, our results lead to a rigorous proof that for a linearly separable problem, AdaBoost with ɛ→0 step-size becomes an L1-margin maximizer when left to run to convergence.

Early stopping is an extremely common tool to minimize overfitting, which would otherwise be a cause of poor generalization of the model to novel data. However, early stopping is a heuristic that, while effective, primarily relies on ad hoc parameters and metrics. Optimizing when to stop remains a challenge. In this paper, we suggest that for some biomedical applications, a natural dichotomy of invasive/non-invasive measurements, or more generally proximal vs distal measurements of a biological system can be exploited to provide objective advice on early stopping. We discuss the conditions where invasive measurements of a biological process should provide better predictions than non-invasive measurements, or at best offer parity. Hence, if data from an invasive measurement are available locally, or from the literature, that information can be leveraged to know with high certainty whether a model of non-invasive data is overfitted. We present paired invasive/non-invasive cardiac and coronary artery measurements from two mouse strains, one of which spontaneously develops type 2 diabetes, posed as a classification problem. Examination of the various stopping rules shows that generalization is reduced with more training epochs and commonly applied stopping rules give widely different generalization error estimates. The use of an empirically derived training ceiling is demonstrated to be helpful as added information to leverage early stopping in order to reduce overfitting.

Tuberculosis (TB) is an infectious disease caused by Mycobacterium tuberculosis, primarily affecting the lungs. Despite being preventable and curable, TB remains a significant global health issue, especially in developing countries. The success of TB treatment heavily depends on the accuracy of the diagnosis, which typically requires expertise from pulmonology or radiology specialists to interpret chest X-ray images. This study aims to design an assistive tool for TB detection that can automatically diagnose the disease using chest X-ray data. The study implemented a Convolutional Neural Network (CNN) architecture to analyze the X-ray images. Additionally, image preprocessing and early stopping methods were employed to enhance accuracy performance, optimize computation, and prevent overfitting. Experiment was conducting using 75% of the data as training data to generate the model and then applied to 25% of the data as testing data. This study comparing image sizes in RGB and grayscale modes. Experimental results show that the use of early stopping has a significant impact on training time, reducing training time substantially in almost all scenarios without drastically sacrificing accuracy. Without early stopping, accuracy does tend to be higher, as seen in grayscale color mode with an image size of 128x128, where the accuracy reaches 0.992, and in RGB mode with an image size of 64x64 which reaches 0.995. However, training time also increases significantly, for example for a 299x299 image with RGB mode, the training time reaches 927 seconds. Therefore, while RGB yields slightly higher accuracy, grayscale is recommended due to significantly faster training times. Additionally, the early stopping mechanism proves effective in reducing computational time, making the training process more efficient.

Automatic early stopping using cross validation: quantifying the criteria

We consider estimators obtained by iterates of the conjugate gradient (CG) algorithm applied to the normal equation of prototypical statistical inverse problems. Stopping the CG algorithm early induces regularisation, and optimal convergence rates of prediction and reconstruction error are established in wide generality for an ideal oracle stopping time. Based on this insight, a fully data-driven early stopping rule τis constructed, which also attains optimal rates, provided the error in estimating the noise level is not dominant. The error analysis of CG under statistical noise is subtle due to its nonlinear dependence on the observations. We provide an explicit error decomposition and identify two terms in the prediction error, which share important properties of classical bias and variance terms. Together with a continuous interpolation between CG iterates, this paves the way for a comprehensive error analysis of early stopping. In particular, a general oracle-type inequality is proved for the prediction error at τ. For bounding the reconstruction error, a more refined probabilistic analysis, based on concentration of self-normalised Gaussian processes, is developed. The methodology also provides some new insights into early stopping for CG in deterministic inverse problems. A numerical study for standard examples shows good results in practice for early stopping at τ.

The large amount of Internet traffic infected with malware disrupts Internet activity. Undetected malwares can shutdown systems and leak sensitive data. To mitigate such threats, a reliable classification model is required. The Compact Convolutional Transformer (CCT) is a deep learning architecture that combines encoder layers from the Vision Transformer with traditional convolutional layers, and is designed to perform classification effectively even with relatively small datasets. However, when tested using a combination of the MalImg dataset and benign class from DikeDataset, the original CCT model showed signs of overfitting. To address this, this study proposes a modification to the CCT architecture by incorporating an average pooling mechanism parallel to the existing sequence pooling layer. The outputs of both pooling layers are concatenated before being passed to the classification layer. Experiments were conducted under several conditions, including full training without early stopping. The results show that the modified CCT with average pooling reduces overfitting and improves accuracy, indicated by a longer training duration before convergence (8+ epochs), an increase in test accuracy by 3.35 with early stopping and by 5.72% without early stopping. Furthermore, this performance improvement was statistically validated using Welch’s t-test on evaluation accuracy with p value 5.83×10⁻⁶ and average validation loss with p value 7.54×10⁻⁷, both yielding significant results (p < 0.05). However, after stratification and class weighted is applied, the evaluation accuracy of modified CCT decrease by 1.74% from the baseline and the validation loss increasing significantly with p-value 0.0009 (p < 0.05), showing that the result is sensitive to the dataset balance.

The development of hyperparameter optimization (HPO) algorithms is an important topic within both the machine learning and data management domains. While numerous strategies employing early stopping mechanisms have been proposed to bolster HPO efficiency, there remains a notable deficiency in understanding how the selection of early stopping metrics influences the reliability of early stopping decisions and, by extension, the broader HPO outcomes. This paper undertakes a systematic exploration of the impact of metric selection on the effectiveness of early stopping-based HPO. Specifically, we introduce a set of metrics that incorporate uncertainty and highlight their practical significance in enhancing the reliability of early stopping decisions. Our empirical experiments on HPO and NAS benchmarks show that using training loss as an early stopping metric in the early training stages improves HPO outcomes by up to 24.76% compared to the more widely accepted validation loss. Furthermore, integrating uncertainty into the metric yields an additional improvement of up to 4% under budget constraints, translating into meaningful resource savings and scalability benefits in large-scale HPO scenarios. These findings demonstrate the critical role of metric selection while shedding light on the potential implications of integrating uncertainty as a metric. This research provides empirical insights that serve as a compass for the selection and formulation of metrics, thereby contributing to a more profound comprehension of mechanisms underpinning early stopping-based HPO.

Introduction: Natural Language Processing (NLP) has experienced significant advancements to address the growing demand for efficient and accurate text classification. Despite numerous methodologies, achieving a balance between high accuracy and model stability remains a critical challenge. This research aims to explore the implementation of a hybrid architecture integrating Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) with FastText embeddings, targeting effective text classification.Methods: The proposed hybrid architecture combines the CNN's ability to capture local patterns and BiLSTM's temporal feature extraction capabilities, enhanced by FastText embeddings for richer word representation. Regulatory mechanisms such as Dropout and Early Stopping were employed to mitigate overfitting. Comparative experiments were conducted to evaluate the performance of the model with and without Early Stopping.Results: The experimental findings reveal that the model without Early Stopping achieved a remarkable accuracy of 99%, albeit with a higher susceptibility to overfitting. Conversely, the implementation of Early Stopping resulted in a stable accuracy of 73%, demonstrating enhanced generalization capabilities while preventing overfitting. The inclusion of Dropout further contributed to model regularization and stability.Conclusions: This study underscores the significance of balancing accuracy and stability in deep learning models for text classification. The proposed hybrid architecture effectively combines the strengths of CNN, BiLSTM, and FastText embeddings, providing valuable insights into the trade-offs between achieving high accuracy and ensuring robust generalization. Future work could further explore optimization techniques and datasets for broader applicability.

Early stopping techniques can be utilized to decrease the time cost, however currently the ultimate goal of early stopping techniques is closely related to the accuracy upgrade or the ability of the neural network to generalize better on unseen data without being large or complex in structure and not directly with its efficiency. Time efficiency is a critical factor in neural networks, especially when dealing with the segmentation of 3D point cloud data, not only because a neural network itself is computationally expensive, but also because point clouds are large and noisy data, making learning processes even more costly. In this paper, we propose a new early stopping technique based on fundamental mathematics aiming to upgrade the trade-off between the learning efficiency and accuracy of neural networks dealing with 3D point clouds. Our results show that by employing our early stopping technique in four distinct and highly utilized neural networks in segmenting 3D point clouds, the training time efficiency of the models is greatly improved, with efficiency gain values reaching up to 94\%, while the models achieving in just a few epochs approximately similar segmentation accuracy metric values like the ones that are obtained in the training of the neural networks in 200 epochs. Also, our proposal outperforms four conventional early stopping approaches in segmentation accuracy, implying a promising innovative early stopping technique in point cloud segmentation.

Early stopping, which is widely used to prevent overfitting, is generally based on a separate validation set. However, in low resource settings, validation-based stopping can be risky because a small validation set may not be sufficiently representative, and the reduction in the number of samples by validation split may result in insufficient samples for training. In this study, we propose an early stopping method that uses unlabeled samples. The proposed method is based on confidence and class distribution similarities. To further improve the performance, we present a calibration method to better estimate the class distribution of the unlabeled samples. The proposed method is advantageous because it does not require a separate validation set and provides a better stopping point by using a large unlabeled set. Extensive experiments are conducted on five text classification datasets and several stop-methods are compared. Our results show that the proposed model even performs better than using an additional validation set as well as the existing stop-methods, in both balanced and imbalanced data settings. Our code is available at https://github.com/DMCB-GIST/BUS-stop.

Plant disease detection is a critical phenomenon, which may enable farmers to reduce losses by increasing productivity. The ability of Convolutional Neural Networks (CNNs) is still limited when dealing with small datasets, which impacts their broader application. A model is prone to overfitting when dealing with a few samples. This limits generalization and reduces its performance since the model may be memorizing data instead of learning general patterns. We used the regularization technique known as early stopping to avoid overfitting and obtain a better generalization. Early stopping ensures the model to generalize and perform well on new data. The EfficientNetB0 architecture — one of the highly advanced CNN architectures — to achieve greater accuracy with fewer parameters. Coupled with the early stopping technique integrated into our approach, to achieve improved generalization for small and diverse datasets.

Ho–Kashyap classifier with early stopping for regularization

Lan and DeMets (1) introduced a flexible procedure for the analysis of sequential trials based on the discretization of the Brownian motion. In this paper, we consider an extension of this strategy that preserves both the desired significance level and the power of any group sequential trial. We propose a procedure that allows for any number and timing of interim analyses. This entails the derivation of boundaries at the monitoring stage by means of two spending functions, one for the type I and one for the type II error probabilities, as well as the adjustment of the target maximum information as the trial progresses. The general solution to the problem is provided together with a discussion of implementation strategies. The procedure is intended for group sequential designs that allow early stopping in favor of both the null and the alternative hypotheses, and an example is presented for this case. However, its application is also easily extended for designs where there is no early stopping in favor of the null.

Early stopping and weight decay are studied in a linear perceptron using a new simplified approach to the dynamics in the thermodynamical limit. The approach is directly deduced from the gradient descent weight update. It allows an exact description of the dynamics of the batch training process. The results are compared with a recent study of early stopping and weight decay based on the equilibrium statistical mechanics approach. It is shown that the equilibrium results for early stopping are good approximations and are exact for weight decay. Furthermore, in the dynamical approach it is possible to determine the necessary number of training steps to fulfil certain termination conditions. It can be shown that asymptotically, i.e. if the number of examples is large, only two batch steps are required to reach the optimal convergence, if the learning rate is optimally chosen.