• Home
  • Search
  • A General Dynamic Knowledge Distillation Method for Visual Analytics.
  • Cite Icon57
  • https://doi.org/10.1109/tip.2022.3212905Copy DOI Icon

A General Dynamic Knowledge Distillation Method for Visual Analytics.

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Existing knowledge distillation (KD) method normally fixes the weight of the teacher network, and uses the knowledge from the teacher network to guide the training of the student network no-ninteractively, thus it is called static knowledge distillation (SKD). SKD is widely used in model compression on the homologous data and knowledge transfer on the heterogeneous data. However, the teacher network that with fixed-weight constrains the student network to learn knowledge from it. It is worth expecting that the teacher network itself can be continuously optimized to promote the learning ability of the student network dynamically. To overcome this limitation, we propose a novel dynamic knowledge distillation (DKD) method, in which the teacher network and the student network can learn from each other interactively. Importantly, we analyzed the effectiveness of DKD mathematically (see Eq. 4), and addressed one crucial issue caused by the continuous change of the teacher network in the dynamic distillation process via designing a valid loss function. We verified the practicality of our DKD by extensive experiments on various visual tasks, e.g. for model compression, we conducted experiments on image classification and object detection. For knowledge transfer, video-based human action recognition is chosen for analysis. The experimental results on benchmark datasets (i.e. ILSVRC2012, COCO2017, HMDB51, UCF101) demonstrated that the proposed DKD is valid to improve the performance of these visual tasks for a large margin. The source code is publicly available online at1.

Similar Papers
  • Research Article
  • Citations6

Discretization and decoupled knowledge distillation for arbitrary oriented object detection

  • Apr 17, 2024
  • Digital Signal Processing
  • Cheng Chen +2
  • Research Article
  • Citations53

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

  • Jan 10, 2022
  • Knowledge-Based Systems
  • Tingsong Ma +2
  • PDF
  • Research Article
  • Citations49

GAN-Knowledge Distillation for One-Stage Object Detection

  • Jan 01, 2020
  • IEEE Access
  • Wanwei Wang +3
  • Research Article
  • Citations31

Skill-Transferring Knowledge Distillation Method

  • Nov 01, 2023
  • IEEE Transactions on Circuits and Systems for Video Technology
  • Shunzhi Yang +5
  • PDF
  • Research Article
  • Citations68

Teacher-student collaborative knowledge distillation for image classification

  • May 04, 2022
  • Applied Intelligence
  • Chuanyun Xu +5
  • Research Article
  • Citations102

Feature Map Distillation of Thin Nets for Low-Resolution Object Recognition.

  • Jan 01, 2022
  • IEEE Transactions on Image Processing
  • Zhenhua Huang +5
  • Research Article
  • Citations38

Data-free knowledge distillation in neural networks for regression

  • Mar 06, 2021
  • Expert Systems with Applications
  • Myeonginn Kang +1
  • Research Article

Cross-modal contrastive learning-based object detection under incomplete modalities

  • Apr 05, 2026
  • Geo-spatial Information Science
  • Hongjun Ma +4
  • Research Article
  • Citations6

PURF: Improving teacher representations by imposing smoothness constraints for knowledge distillation

  • Apr 09, 2024
  • Applied Soft Computing
  • Md Imtiaz Hossain +3
  • Research Article
  • Citations11

Efficient face anti-spoofing via head-aware transformer based knowledge distillation with 5 MB model parameters

  • Sep 10, 2024
  • Applied Soft Computing
  • Jun Zhang +6
  • Research Article
  • Citations64

Low-light image enhancement with knowledge distillation

  • Nov 11, 2022
  • Neurocomputing
  • Ziwen Li +2
  • Research Article

Looking Broader for Knowledge Distillation Via Receptive-Field Alignment.

  • Jun 17, 2026
  • IEEE transactions on pattern analysis and machine intelligence
  • Sihao Lin +7
  • Research Article
  • Citations12

Multi-perspective analysis on data augmentation in knowledge distillation

  • Mar 05, 2024
  • Neurocomputing
  • Wei Li +3
  • Conference Article
  • Citations3

Plant Leaf Segmentation Using Knowledge Distillation

  • Oct 26, 2022
  • Joo-Yeon Jung +2
  • Research Article
  • Citations83

Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method.

  • Jun 01, 2024
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Shunzhi Yang +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.