Mitigating Catastrophic Forgetting In Adaptive Class Incremental Extreme Learning Machine Through Neuron Clustering

Catastrophic forgetting in neural networks during incremental learning remains a challenging problem. Previous research investigated the catastrophic forgetting in fully connected networks with some earlier work exploring activation functions and learning algorithms. Applications of neural networks have been extended to include similarity and metric learning. It is of significant interest to understand how metric learning loss functions would be affected by catastrophic forgetting. Our research investigates catastrophic forgetting for four well-known metric-based loss functions during incremental class learning. The loss functions are angular, contrastive, center, and triplet loss. Our results show that the rate of forgetting is different across loss functions on multiple datasets. Triplet loss was least affected followed by contrastive, center, and angular loss. Center and angular loss produce better embeddings on difficult tasks when trained on all available training data, however, they are the least robust to forgetting during incremental class learning. We argue that triplet loss provides the ideal middle ground for future improvements.

Many deep learning applications, like keyword spotting, require the incorporation of new concepts (classes) over time, referred to as Class Incremental Learning (CIL). The major challenge in CIL is catastrophic forgetting, i.e., preserving as much of the old knowledge as possible while learning new tasks. Various techniques, such as regularization, knowledge distillation, and the use of exemplars, have been proposed to resolve this issue. However, prior works primarily focus on the incremental learning step, while ignoring the optimization during the base model training. We hypothesize that a more transferable and generalizable feature representation from the base model would be beneficial to incremental learning. In this work, we adopt multitask learning during base model training to improve the feature generalizability. Specifically, instead of training a single model with all the base classes, we decompose the base classes into multiple subsets and regard each of them as a task. These tasks are trained concurrently and a shared feature extractor is obtained for incremental learning. We evaluate our approach on two datasets under various configurations. The results show that our approach enhances the average incremental learning accuracy by up to 5.5%, which enables more reliable and accurate keyword spotting over time. Moreover, the proposed approach can be combined with many existing techniques and provides additional performance gain.

Artificial neural networks have been reported to exhibit, and in some cases surpass, human level performance on individual rigid tasks. However, these networks remain static entities of knowledge for those specific tasks, which can lead to catastrophic forgetting ---i.e., forgetting old tasks--- when attempting to learn new tasks. The main objective of Incremental Learning (IL) is to address this issue. In order to tackle catastrophic forgetting, various approaches have been proposed so far. We survey those approaches and organize them in nine categories: regularization-based methods, exemplar replay-based methods, variational continual learning-based methods, parameter isolation-based methods, dynamic architectures-based methods, distillation-based methods, generative methods, data-free methods and unsupervised methods. Moreover, this review distinguishes between two scenarios, Task Incremental Learning (Task-IL) and Class Incremental Learning (Class-IL), and reports on the results obtained for a number of experiments that compare the performance achieved by a selection of diverse methods for both scenarios on the datasets most used by the related research community.

With the advancement and widespread adoption of deep learning models, there has been a growing interest in class incremental learning. This approach aims to continuously learn new classes while retaining the recognition and memory capabilities for previously learned classes within an open and dynamic environment. The primary focus of class incremental learning is on maintaining the ability to learn new classes while mitigating catastrophic forgetting, thus achieving a better balance between stability and adaptability. To address this challenge, we propose an innovative method for incremental class learning that leverages dynamically representations to facilitate more efficient incremental class learning, preserving previously acquired features while adapting to new ones and effectively reducing catastrophic forgetting. Furthermore, we introduce a feature augmentation mechanism to significantly enhance the model's classification performance when incorporating new classes. This approach ensures efficient learning of both old and new classes without compromising the effectiveness of previous models. We conducted extensive experiments on two classes incremental learning benchmarks, consistently demonstrating significant performance advantages over other methods.

The development of efficient arrhythmia detection systems is crucial for physiological measurements and computer-aided diagnosis. Existing systems rely mainly on offline learning methods and lack the ability to assimilate new data or recognize emerging categories. To address these challenges, this study introduces an arrhythmia detection model that is resistant to data imbalance and has continuous learning capabilities, specifically for incremental learning of new ECG data categories. The system incorporates constraints derived from the new class data and implements a dynamic mechanism for updating connection weights, facilitating the incremental continual learning of classes. Confronted with the problem of models forgetting the original data and overfitting with the added data in continuous learning, we introduce a data balancing method by regularization to balance the model's memory and learning of the two types of data. Facing the data imbalance problem in continuous learning, we introduce a posteriori probability weighting strategy. This strategy assigns greater importance to high-value samples based on the model's posterior residual kernel density estimates. Comprehensive testing of the model using various datasets from the MIT-BIH database indicates superior performance in incremental learning tasks. The results reveal that the model not only excels in class incremental learning but also ensures effective balancing across different data classes.

The ability to learn more concepts from incrementally arriving data over time is essential for the development of a lifelong learning system. However, deep neural networks often suffer from forgetting previously learned concepts when continually learning new concepts, which is known as the catastrophic forgetting problem. The main reason for catastrophic forgetting is that past concept data are not available, and neural weights are changed during incrementally learning new concepts. In this article, we propose an incremental concept learning framework that includes two components, namely, ICLNet and RecallNet. ICLNet, which consists of a trainable feature extractor and a dynamic concept memory matrix, aims to learn new concepts incrementally. We propose a concept-contrastive loss to alleviate the magnitude of neural weight changes and mitigate the catastrophic forgetting problems. RecallNet aims to consolidate old concepts memory and recall pseudo samples, whereas ICLNet learns new concepts. We propose a balanced online memory recall strategy to reduce the information loss of old concept memory. We evaluate the proposed approach on the MNIST, Fashion-MNIST, and SVHN data sets and compare it with other pseudorehearsal-based approaches. Extensive experiments demonstrate the effectiveness of our approach.

Few-shot class incremental learning is a challenging problem in the field of machine learning. It necessitates models to gradually learn new knowledge from a few samples while retaining the knowledge of old classes. Nevertheless, the limited data available for new classes not only leads to significant overfitting problems but also exacerbates the issue of catastrophic forgetting in the incremental learning process. To address the above two issues, we propose a novel framework named Grassmann Manifold and Information Entropy for Few-Shot Class Incremental Learning(GMIE-FSCIL). Different from existing methods that model parameters on the Euclidean space, our method optimizes the incremental learning network on the Grassmann manifold. More specifically, we incorporate the acquired knowledge of each class on the Grassmann manifold, ensuring the preservation of their inherent geometric properties by Grassmann Metric Learning(GML) module. Acknowledging the interconnected relationships of knowledge, with information entropy we create a neighborhood graph on Grassmann manifold to maintain inter-class structural information by Graph Information Preserving(GIP) module, thus mitigating catastrophic forgetting of learned knowledge. In our evaluation of CIFAR100, miniImageNet, and CUB200 datasets, we achieved significant improvements in terms of Avg compared to mainstream methods, with at least 2.72%, 1.21%, and 1.27% increases.

Class incremental learning with analytic learning for hyperspectral image classification

Catastrophic forgetting is a prominent challenge in machine learning. It denotes the phenomenon wherein models undergo a significant loss of previously acquired knowledge upon learning new information. Supervised Continual Learning (SCL) has emerged as a promising approach to mitigate this issue by enabling models to adapt to non-stationary data distributions while leveraging labeled data. However, practical limitations arise for SCL in real-world settings, where labeled data is scarce. Conversely, Unsupervised Continual Learning (UCL) aims to mitigate forgetting without the need for manual annotations. Still, many previous UCL methods rely on replay-based strategies, which may not be viable in contexts where storing training data is impractical. Additionally, replay methods may face challenges related to overfitting and representation drifts when the buffer size is limited. To overcome the limitations of replay strategies, we propose an approach based on parameter isolation, combined with gradient projection. Specifically, we use task-specific hard attention and gradient projection to constrain updates to parameters crucial for previous tasks. Furthermore, our approach offers advantages over architecture-based methodologies by avoiding the need for network expansion and allowing for sequential learning within a predefined network architecture. Contrastive learning-based unsupervised methods are effective in preserving representation continuity. However, the loss in contrastive learning may suffer from a decrease in the diversity of negative samples. To address this, we incorporate both direct instance grouping and discrimination at the cross-level for actual and pseudo groups to contrastively learn unsupervised representations.We employ evidential deep learning (EDL) on rotation augmentation-based pseudo groups to effectively identify OOD instances and learn distinct class representations. Through comprehensive experiments, we demonstrate that the proposed model achieves an overall-average accuracy of 77.82% for task incremental learning (TIL), and 67.39% for class incremental learning (CIL) across various benchmark datasets, encompassing both short and long sequences of tasks. Notably, our model achieves almost zero forgetting, outperforming state-of-the-art (SOTA) baseline accuracies of 73.33% and 60.8% for TIL and CIL, respectively. Additionally, our model demonstrates a significant improvement over the SOTA SCL baseline, achieving a 2.82% and 3.68% increase in average TIL and CIL accuracy, respectively, while substantially reducing forgetting from approximately 12.66% and 19.56% to nearly zero.

Extreme learning machine (ELM) is a learning algorithm for single-hidden layer feedforward neural networks. In theory, this algorithm is able to provide good generalization capability at extremely fast learning speed. Comparative studies of benchmark function approximation problems revealed that ELM can learn thousands of times faster than conventional neural network (NN) and can produce good generalization performance in most cases. Unfortunately, the research on damage localization using ELM is limited in the literature. In this chapter, the ELM is extended to the domain of damage localization of plate structures. Its effectiveness in comparison with typical neural networks such as back-propagation neural network (BPNN) and least squares support vector machine (LSSVM) is illustrated through experimental studies. Comparative investigations in terms of learning time and localization accuracy are carried out in detail. It is shown that ELM paves a new way in the domain of plate structure health monitoring. Both advantages and disadvantages of using ELM are discussed.

Class-incremental learning (CIL) aims to incrementally learn a unified classifier for new classes emerging, which suffers from the catastrophic forgetting problem. To alleviate forgetting and improve the recognition performance, we propose a novel CIL framework, named the topological schemas model (TSM). TSM consists of a Gaussian mixture model arranged on 2D grids (2D-GMM) as the memory of the learned knowledge. To train the 2D-GMM model, we develop a novel competitive expectation-maximization (CEM) method, which contains a global topology embedding step and a local expectation-maximization fine-tuning step. Meanwhile, we choose the image samples of old classes that have the maximum posterior probability with respect to each Gaussian distribution as the episodic points. When finetuning for new classes, we propose the memory preservation loss (MPL) term to ensure episodic points still have maximum probabilities with respect to the corresponding Gaussian distribution. MPL preserves the distribution of 2D-GMM for old knowledge during incremental learning and alleviates catastrophic forgetting. Comprehensive experimental evaluations on two popular CIL benchmarks CIFAR100 and subImageNet demonstrate the superiority of our TSM.

Deep neural networks can be used to perform nonlinear operations at multiple levels, such as a neural network that is composed of many hidden layers. Although deep learning approaches show good results, they have a drawback called catastrophic forgetting, which is a reduction in performance when a new class is added. Incremental learning is a learning method where existing knowledge should be retained even when new data is acquired. It involves learning with multiple batches of training data and the newer learning sessions do not require the data used in the previous iterations. The Bayesian approach to incremental learning uses the concept of the probability distribution of weights. The key idea of Bayes theorem is to find an updated distribution of weights and biases. In the Bayesian framework, the beliefs can be updated iteratively as the new data comes in. Bayesian framework allows to update the beliefs iteratively in real-time as data comes in. The Bayesian model for incremental learning showed an accuracy of 82%. The execution time for the Bayesian model was lesser on GPU (670 s) when compared to CPU (1165 s).

With the exponential growth in the speed and volume of remote sensing data, deep learning models are expected to adapt and continually learn over time. Unfortunately, the domain shift between multi-source remote sensing data from various sensors and regions poses a significant challenge. Segmentation models face difficulty in adapting to incremental domains due to catastrophic forgetting, which can be addressed via incremental learning methods. However, current incremental learning methods mainly focus on class-incremental learning, wherein classes belong to the same remote sensing domain, and neglect investigations into incremental domains in remote sensing. To solve this problem, we propose a domain-incremental learning method for semantic segmentation in multi-source remote sensing data. Specifically, our model aims to incrementally learn a new domain while preserving its performance on previous domains without accessing previous domain data. To achieve this, our model has a unique parameter learning structure that reparametrizes domain-agnostic and domain-specific parameters. We use different optimization strategies to adapt to domain shift in incremental domain learning. Additionally, we adopt multi-level knowledge distillation loss to mitigate the impact of label space shift among domains. The experiments demonstrate that our method achieves excellent performance in domain-incremental settings, outperforming existing methods with only a few parameters.

Extreme learning machine (ELM) is a learning algorithm for generalized single-hidden-layer feed-forward networks (SLFNs). In order to obtain a suitable network architecture, Incremental Extreme Learning Machine (I-ELM) is a sort of ELM constructing SLFNs by adding hidden nodes one by one. Although kinds of I-ELM-class algorithms were proposed to improve the convergence rate or to obtain minimal training error, they do not change the construction way of I-ELM or face the over-fitting risk. Making the testing error converge quickly and stably therefore becomes an important issue. In this paper, we proposed a new incremental ELM which is referred to as Length-Changeable Incremental Extreme Learning Machine (LCI-ELM). It allows more than one hidden node to be added to the network and the existing network will be regarded as a whole in output weights tuning. The output weights of newly added hidden nodes are determined using a partial error-minimizing method. We prove that an SLFN constructed using LCI-ELM has approximation capability on a universal compact input set as well as on a finite training set. Experimental results demonstrate that LCI-ELM achieves higher convergence rate as well as lower over-fitting risk than some competitive I-ELM-class algorithms.

Class-incremental learning (CIL) in deep neural networks is affected by catastrophic forgetting (CF), where acquiring knowledge of new classes leads to the significant degradation of previously learned representations. This challenge is particularly severe in medical image analysis, where costly, expertise-dependent annotations frequently contain pervasive and hard-to-detect noisy labels that substantially compromise model performance. While existing approaches have predominantly addressed CF and noisy labels as separate problems, their combined effects remain largely unexplored. To address this critical gap, this paper presents a dual-stage clean-sample selection method for Incremental Noisy Label Learning (DSCNL). Our approach comprises two key components: (1) a dual-stage clean-sample selection module that identifies and leverages high-confidence samples to guide the learning of reliable representations while mitigating noise propagation during training, and (2) an experience soft-replay strategy for memory rehearsal to improve the model's robustness and generalization in the presence of historical noisy labels. This integrated framework effectively suppresses the adverse influence of noisy labels while simultaneously alleviating catastrophic forgetting. Extensive evaluations on public medical image datasets demonstrate that DSCNL consistently outperforms state-of-the-art CIL methods across diverse classification tasks. The proposed method boosts the average accuracy by 55% and 31% compared with baseline methods on datasets with different noise levels, and achieves an average noise reduction rate of 73% under original noise conditions, highlighting its effectiveness and applicability in real-world medical imaging scenarios.