Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning

Many deep learning applications, like keyword spotting, require the incorporation of new concepts (classes) over time, referred to as Class Incremental Learning (CIL). The major challenge in CIL is catastrophic forgetting, i.e., preserving as much of the old knowledge as possible while learning new tasks. Various techniques, such as regularization, knowledge distillation, and the use of exemplars, have been proposed to resolve this issue. However, prior works primarily focus on the incremental learning step, while ignoring the optimization during the base model training. We hypothesize that a more transferable and generalizable feature representation from the base model would be beneficial to incremental learning. In this work, we adopt multitask learning during base model training to improve the feature generalizability. Specifically, instead of training a single model with all the base classes, we decompose the base classes into multiple subsets and regard each of them as a task. These tasks are trained concurrently and a shared feature extractor is obtained for incremental learning. We evaluate our approach on two datasets under various configurations. The results show that our approach enhances the average incremental learning accuracy by up to 5.5%, which enables more reliable and accurate keyword spotting over time. Moreover, the proposed approach can be combined with many existing techniques and provides additional performance gain.

Modern artificial intelligence systems require class-incremental learning while suffering from catastrophic forgetting in many real-world applications. Due to the missing knowledge of past data, performance substantially degrades. Recent methods often used knowledge distillation and bias correction to avoid catastrophic forgetting caused by cognitive bias. However, since these methods mainly learn all samples indiscriminately, the model is hard to learn what it truly needs from the data stream to balance the new and old knowledge, leading to inevitable forgetting. Instead of considering each sample indiscriminately, the model should learn from its curious samples automatically. To tackle this problem, we propose a curiosity-driven class-incremental learning approach via adaptive sample selection for learning a more generalized model with fewer ineffective updates. Specifically, our method quantifies the model’s curiosity in each sample by two properties: uncertainty and novelty. Our model learns informative samples selectively during training utilizing the proposed uncertainty property, which benefits the classification decision boundary. In light of the imbalanced data, a novelty property is used to selectively optimize the model by employing dissimilar samples, endowing it with more robustness and less cognitive bias. Our method successfully reduces catastrophic forgetting and can be flexibly incorporated with other techniques. Extensive experiments and in-depth analysis on the CIFAR-100, Tiny-ImageNet and Caltech-101 datasets show that our approach outperforms competing methods for class-incremental learning in terms of preventing catastrophic forgetting.

In class-incremental learning setting, classes are typically presented batch by batch over time. Incremental learning often suffers from catastrophic forgetting: the performance on previous classes abruptly degrades when adapting a model to new classes. We find that incremental models trained using knowledge distillation are skilled at discriminating classes within a batch, whereas they have confusion among classes in different batches. We propose a class-incremental learning approach with knowledge amalgamation and batch identification (KABI), which can effectively alleviate catastrophic forgetting. The idea is to first train an expert model for new classes at current state, and then train an amalgamation model by amalgamating knowledge from the expert model and the amalgamation model of previous state to discriminate different classes within a batch, and particularly train a batch identifier to discriminate different batches. We conduct extensive experiments on three datasets: MNIST, CIFAR-100, ILSVRC 2012, and show that KABI outperforms the second-best approach by 1.29%, 14.26% and 21.51% respectively. Surprisingly, classification accuracies of our approach are even sometimes higher than the oracle results which is obtained by training a model using all training samples from all classes.

Specific emitter identification (SEI) is a non-cryptographic authentication technique to provide an extra security layer for wireless devices, which has promising applications. However, the traditional methods of SEI are only available in limited equipments. In actual application scenarios, new devices (as new classes) are constantly appearing. In this paper, an effective class incremental learning (CIL) method is proposed for SEI, named class-incremental with self-training and prototype augmentation (CISP). It is a teacher-student network. Firstly, the teacher network trained by the old-class data is utilized to instruct the student network to adapt the new classes while retaining the old-class knowledge through the knowledge distillation (KD) techniques. Secondly, in order to mitigate the problem of favoring the new classes, weight aligning (WA) method is introduced to balance the weights of the new-class and old-class classification layers in the student network. Lastly, the old-class samples are recalled from the unlabeled dataset by the student network and input into the teacher network. Then the feature prototypes of the old classes are constructed and augmented. This would further ease the imbalance between the old and new classes and alleviate the problem of noisy pseudo-labels. Experiment results on the real AIS-100 dataset and ADS-B-100 dataset with the number of the initial classes being 20 and 20 classes per incremental step demonstrate that the proposed method can achieve an average accuracy of 95.29% and 95.84%, respectively. It effectively mitigates the catastrophic forgetting of the model and is superior to the state-of-the-art incremental learning approaches of not saving the old-class samples.

Few-shot class-incremental learning (FSCIL) has been proposed aiming to enable a deep learning system to incrementally learn new classes with limited data. Recently, a pioneer claims that the commonly used replay-based method in class-incremental learning (CIL) is ineffective and thus not preferred for FSCIL. This has, if truth, a significant influence on the fields of FSCIL. In this paper, we show through empirical results that adopting the data replay is surprisingly favorable. However, storing and replaying old data can lead to a privacy concern. To address this issue, we alternatively propose using data-free replay that can synthesize data by a generator without accessing real data. In observing the effectiveness of uncertain data for knowledge distillation, we impose entropy regularization in the generator training to encourage more uncertain examples. Moreover, we propose to relabel the generated data with one-hot-like labels. This modification allows the network to learn by solely minimizing the cross-entropy loss, which mitigates the problem of balancing different objectives in the conventional knowledge distillation approach. Finally, we show extensive experimental results and analysis on CIFAR-100, miniImageNet and CUB-200 to demonstrate the effectiveness of our proposed one.

Class-incremental learning (CIL) enables models to continuously acquire knowledge and adapt in an ever-changing environment. However, one primary challenge lies in the trade-off between the stability and plasticity, i.e., plastically expand the novel knowledge base and stably retaining previous knowledge without catastrophic forgetting. We find that even recent promising CIL methods via pre-trained models (PTMs) still suffer from this dilemma. To this end, this paper begins by analyzing the aforementioned dilemma from the perspective of marginal distribution for data categories. Then, we propose the feature transformation tuning (FeTT) model, which concurrently alleviates the inadequacy of previous PTM-based CIL in terms of stability and plasticity. Specifically, we apply the parameter-efficient fine-tuning (PEFT) strategies solely in the first CIL task to bridge the domain gap between the PTMs and downstream task dataset. Subsequently, the model is kept fixed to maintain stability and avoid discrepancies in training data distributions. Moreover, feature transformation is employed to regulate the backbone representations, boosting the model’s adaptability and plasticity without additional training or parameter costs. Extensive experimental results and further feature channel activations discussion on CIL benchmarks across six datasets validate the superior performance of our proposed method.

Class incremental learning via dynamic regeneration with task-adaptive distillation

Knowledge distillation (KD) aims to distill the knowledge from the teacher (larger) to the student (smaller) model via soft-label for the efficient neural network. In general, the performance of a model is determined by accuracy, which is measured with labels. However, existing KD approaches usually use the teacher with its original distribution, neglecting the potential of incorrect prediction. This may contradict the motivation of hard-label learning through cross-entropy loss, which may lead to sub-optimal knowledge distillation on certain samples. To address this issue, we propose a novel logit processing scheme via a sorting mechanism. Specifically, our method has a two-fold goal: (1) fixing the incorrect prediction of the teacher based on the labels and (2) reordering the distribution in a natural way according to priority rank at once. As an easy-to-use, plug-andplay pre-processing, our sort method can be effectively applied to existing logit-based KD methods. Extensive experiments on the CIFAR-100 and ImageNet datasets demonstrate the effectiveness of our method.

We address an important and practical problem facing deep-learning-based image classification: class incremental learning with a large domain shift. Most previous efforts on class incremental learning focus on one aspect of the problem, i.e., learning to classify additional new classes (with a little shift). However, in the real world, when new classes are added, the domain changes simultaneously (with a large domain shift). To obtain a model that is robust to these situations, we need to consider incrementally learning not only new labels but also domain-shifted labels. We target a continual and simultaneous shift of class and domain distribution and propose a new incremental learning method named Momentum Contrastive learning enhancing Orthogonality of Negative pairs (MoCo-ON). We employ a momentum encoder framework augmented with rehearsal memory to mitigate the risk of forgetting while leveraging contrastive learning to extract versatile features capable of adapting to the progressively shifting domain. Specifically, when training with a knowledge distillation loss, we introduce a novel supervised contrastive loss designed to closely embed positive pairs of the same class, even in the presence of a substantial domain gap. Additionally, we leverage feature embedding from momentum encoders for exemplar selection, aiming to mitigate the risk of forgetting previously acquired information from earlier tasks. We conduct comprehensive experiments involving inter-domain shifted class incremental learning scenarios using widely adopted datasets commonly employed for studying domain generalization in image classification. Our proposed method consistently outperforms other methods by a significant margin.

With the advancement and widespread adoption of deep learning models, there has been a growing interest in class incremental learning. This approach aims to continuously learn new classes while retaining the recognition and memory capabilities for previously learned classes within an open and dynamic environment. The primary focus of class incremental learning is on maintaining the ability to learn new classes while mitigating catastrophic forgetting, thus achieving a better balance between stability and adaptability. To address this challenge, we propose an innovative method for incremental class learning that leverages dynamically representations to facilitate more efficient incremental class learning, preserving previously acquired features while adapting to new ones and effectively reducing catastrophic forgetting. Furthermore, we introduce a feature augmentation mechanism to significantly enhance the model's classification performance when incorporating new classes. This approach ensures efficient learning of both old and new classes without compromising the effectiveness of previous models. We conducted extensive experiments on two classes incremental learning benchmarks, consistently demonstrating significant performance advantages over other methods.

A novel method for tackling the problem of imbalanced data in medical image segmentation is proposed in this work. In Balanced Cross Entropy (BCE) loss, which is a type of weighted CE loss, the weight assigned to each class is the inverse of the class frequency. These balancing weights are expected to equalize the effect of each class on the overall loss and prevent the model from being biased towards the majority class. But, as it has been shown in previous studies, this method degrades the performance by a large margin. Therefore, Balanced CE is not a popular loss in medical segmentation tasks, and usually a region-based loss, like the Dice loss, is used to address the class imbalance problem. Existing weighting CE loss approaches, like Balanced CE, are usually adapted from classification tasks. In this study we propose a weighting method which is designed for segmentation and takes advantage of spatial and geometrical properties of the labels. In the proposed method, the weighting of cross entropy loss for each class is based on a dilated area of each class mask, and balancing weights are assigned to each class together with its surrounding pixels. The goal of this study is to show that the performance of Balanced CE loss can be greatly improved my modifying its weighting strategy. Experiments on different datasets show that the proposed Dilated Balanced CE (DBCE) loss outperforms the Balanced CE loss by a large margin and produces superior results compared to CE loss, and its performance is similar or better compared to the performance of the combination of Dice and CE loss. Also, the performance of different combination losses can be improved by replacing the CE loss with the proposed DBCE loss. This study shows that a weighted cross entropy loss with the right weighing strategy can be as effective as a region-based loss in handling the class imbalance problem.

Catastrophic forgetting in neural networks during incremental learning remains a challenging problem. Previous research investigated the catastrophic forgetting in fully connected networks with some earlier work exploring activation functions and learning algorithms. Applications of neural networks have been extended to include similarity and metric learning. It is of significant interest to understand how metric learning loss functions would be affected by catastrophic forgetting. Our research investigates catastrophic forgetting for four well-known metric-based loss functions during incremental class learning. The loss functions are angular, contrastive, center, and triplet loss. Our results show that the rate of forgetting is different across loss functions on multiple datasets. Triplet loss was least affected followed by contrastive, center, and angular loss. Center and angular loss produce better embeddings on difficult tasks when trained on all available training data, however, they are the least robust to forgetting during incremental class learning. We argue that triplet loss provides the ideal middle ground for future improvements.

In a matter that is analogous to the study of natural image statistics, we study the natural statistics of the deep neural network activations at various layers. As we show, these statistics, similar to image statistics, follow a power law. We also show, both analytically and empirically, that with depth the exponent of this power law increases at a linear rate.As a direct implication of our discoveries, we present a method for performing Knowledge Distillation (KD). While classical KD methods consider the logits of the teacher network, more recent methods obtain a leap in performance by considering the activation maps. This, however, uses metrics that are suitable for comparing images. We propose to employ two additional loss terms that are based on the spectral properties of the intermediate activation maps. The proposed method obtains state of the art results on multiple image recognition KD benchmarks.

Enhancing marine oil spill detection through dynamic adaptive knowledge distillation with spectral mask superpixel.

Hyperspectral image (HSI) classification models suffer from a phenomenon known as catastrophic forgetting, which refers to the sharp decline in performance on previously learned tasks after learning a new one when continuously acquiring new knowledge from a sequence of tasks. In recent years, some lifelong learning approaches have been proposed for HSI classification. Despite some progress, the challenge of catastrophic forgetting in lifelong learning remains significant and unresolved. In this article, we propose a novel lifelong learning framework for HSI classification, which is based on exemplar replay and cross-spectral–spatial feature knowledge distillation (KD), termed HyperKD. Specifically, the proposed framework incorporates a min-max cross-selection (MMCS) module tailored to HSI characteristics with a cross-spectral-spatial knowledge distillation (CSSKD) module. The MMCS module selects the most representative or diverse samples as exemplars from previous tasks for replay. Additionally, the CSSKD module not only transfers the prediction logit distribution from the previous network to the current network but also transfers the spectral-spatial feature distribution via cross-network KD, without directly assessing the similarity of feature distributions, thereby retaining more knowledge and mitigating forgetting. Through experiments conducted on a series of tasks, including the Pavia, Indian Pines, Salinas, and Houston datasets, our approach demonstrates superior performance compared to previous lifelong learning methods for HSI classification, effectively mitigating catastrophic forgetting. The code implementation of our approach will be publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lzlsxs/hyperkd</uri>.