• Home
  • Search
  • Subclass Knowledge Distillation with Known Subclass Labels
  • Cite Icon5
  • https://doi.org/10.1109/ivmsp54334.2022.9816232Copy DOI Icon

Subclass Knowledge Distillation with Known Subclass Labels

  • Jun 26, 2022
  • Ahmad Sajedi +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

This work introduces a novel knowledge distillation framework for classification tasks where information on existing subclasses is available and taken into consideration. In classification tasks with a small number of classes or binary detection, the amount of information transferred from the teacher to the student is restricted, thus limiting the utility of knowledge distillation. Performance can be improved by leveraging information of possible subclasses within the classes. To that end, we propose the so-called Subclass Knowledge Distillation (SKD), a process of transferring the knowledge of predicted subclasses from a teacher to a smaller student. Meaningful information that is not in the teacher’s class logits but exists in subclass logits (e.g., similarities within classes) will be conveyed to the student through the SKD, which will then boost the student’s performance. Analytically, we measure how much extra information the teacher can provide the student via the SKD to demonstrate the efficacy of our work. The framework developed is evaluated in clinical application, namely colorectal polyp binary classification. It is a practical problem with two classes and a number of subclasses per class. In this application, clinician-provided annotations are used to define subclasses based on the annotation label’s variability in a curriculum style of learning. A lightweight, low-complexity student trained with the SKD framework achieves an F1-score of 85.05%, an improvement of 1.47%, and a 2.10% gain over the student that is trained with and without conventional knowledge distillation, respectively. The 2.10% F1-score gap between students trained with and without the SKD can be explained by the extra subclass knowledge, i.e., the extra 0.4656 label bits per sample that the teacher can transfer in our experiment. The SKD framework can benefit from using more information to increase student performance, but it comes at the expense of the availability of subclass labels.

Similar Papers
  • PDF
  • Research Article
  • Citations33

Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification.

  • Jan 01, 2022
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Quanshi Zhang +3
  • Research Article

Distilling Structural Knowledge for Platform-Aware Semantic Segmentation

  • May 01, 2024
  • Journal of Physics: Conference Series
  • Guilin Li +2
  • Research Article

AMAKD: Adversarial multi-modal attention-based knowledge distillation for robust behaviour anomaly detection in real-world environments

  • Apr 01, 2026
  • Alexandria Engineering Journal
  • Maram Fahaad Almufareh
  • Conference Article
  • Citations11

Augmenting Knowledge Distillation with Peer-to-Peer Mutual Learning for Model Compression

  • Mar 28, 2022
  • Usma Niyaz +1
  • Research Article
  • Citations212

Adversarially Robust Distillation

  • Apr 03, 2020
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Micah Goldblum +3
  • Conference Article
  • Citations5

Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution

  • Jun 01, 2021
  • Hadi Pouransari +3
  • Research Article
  • Citations14

Knowledge distillation guided by multiple homogeneous teachers

  • May 31, 2022
  • Information Sciences
  • Quanzheng Xu +2
  • Research Article
  • Citations2

De-biased knowledge distillation framework based on knowledge infusion and label de-biasing techniques

  • Aug 16, 2024
  • Journal of Electronic Science and Technology
  • Yan Li +3
  • Book Chapter
  • Citations19

Self-Referenced Deep Learning

  • Jan 01, 2019
  • Xu Lan +2
  • Research Article

Enhancing end-to-end speech translation via multi-stage knowledge distillation.

  • May 01, 2026
  • Neural networks : the official journal of the International Neural Network Society
  • Yue Zhou +3
  • Research Article
  • Citations10

Adaptive weighted multi-teacher distillation for efficient medical imaging segmentation with limited data

  • Apr 01, 2025
  • Knowledge-Based Systems
  • Eddardaa Ben Loussaief +4
  • Conference Article
  • Citations12

Data-Free Ensemble Knowledge Distillation for Privacy-conscious Multimedia Model Compression

  • Oct 17, 2021
  • Zhiwei Hao +4
  • PDF
  • Research Article
  • Citations36

Knowledge Distillation in Acoustic Scene Classification

  • Jan 01, 2020
  • IEEE Access
  • Jee-Weon Jung +3
  • Research Article
  • Citations1

A Multifunctional Network with Uncertainty Estimation and Attention-Based Knowledge Distillation to Address Practical Challenges in Respiration Rate Estimation

  • Feb 01, 2023
  • Sensors (Basel, Switzerland)
  • Kapil Singh Rathore +3
  • Conference Article

Triplet Knowledge Distillation Networks for Model Compression

  • Jul 18, 2021
  • Jialiang Tang +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.