• Home
  • Search
  • Multi-target Knowledge Distillation via Student Self-reflection
  • Cite Icon52
  • https://doi.org/10.1007/s11263-023-01792-zCopy DOI Icon

Multi-target Knowledge Distillation via Student Self-reflection

Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher’s ability in unfolding the knowledge to be distilled, but also improve the student’s capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.

Loading PDF

Similar Papers
  • Conference Article
  • Citations5

Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

  • May 13, 2024
  • Liyang He +6
  • Research Article
  • Citations6

Discretization and decoupled knowledge distillation for arbitrary oriented object detection

  • Apr 17, 2024
  • Digital Signal Processing
  • Cheng Chen +2
  • Research Article
  • Citations8

Dual Knowledge Distillation for neural machine translation

  • Nov 09, 2023
  • Computer Speech & Language
  • Yuxian Wan +4
  • Research Article
  • Citations6

PURF: Improving teacher representations by imposing smoothness constraints for knowledge distillation

  • Apr 09, 2024
  • Applied Soft Computing
  • Md Imtiaz Hossain +3
  • Conference Article
  • Citations3

Spatial-temporal consistency knowledge distillation for real-time semantic segmentation

  • Nov 19, 2022
  • Dongli Wang +3
  • Research Article
  • Citations2

Feature fusion-based collaborative learning for knowledge distillation

  • Nov 01, 2021
  • International Journal of Distributed Sensor Networks
  • Yiting Li +4
  • Research Article
  • Citations12

Multi-perspective analysis on data augmentation in knowledge distillation

  • Mar 05, 2024
  • Neurocomputing
  • Wei Li +3
  • Research Article
  • Citations33

Resolution-Aware Knowledge Distillation for Efficient Inference.

  • Jan 01, 2021
  • IEEE Transactions on Image Processing
  • Zhanxiang Feng +2
  • Conference Article
  • Citations1

Adversarial Metric Knowledge Distillation

  • Nov 27, 2020
  • Zihe Dong +3
  • Book Chapter
  • Citations9

What Role Does Data Augmentation Play in Knowledge Distillation?

  • Jan 01, 2023
  • Wei Li +5
  • Conference Article
  • Citations15

Triplet Loss for Knowledge Distillation

  • Jul 01, 2020
  • Hideki Oki +3
  • PDF
  • Research Article

Multiloss Joint Gradient Control Knowledge Distillation for Image Classification

  • Oct 17, 2024
  • Electronics
  • Wei He +6
  • Book Chapter
  • Citations18

HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors

  • Jan 01, 2022
  • Luting Wang +7
  • Research Article
  • Citations1

Knowledge distillation of face recognition via attention cosine similarity review

  • May 31, 2024
  • IET Computer Vision
  • Zhuo Wang +2
  • Research Article
  • Citations13

Maximizing discrimination capability of knowledge distillation with energy function

  • May 08, 2024
  • Knowledge-Based Systems
  • Seonghak Kim +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.