• Home
  • Search
  • Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks
  • Cite Icon21
  • https://doi.org/10.1609/aaai.v35i8.16839Copy DOI Icon

Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by randomly changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.

Similar Papers
  • Research Article
  • Citations6

PURF: Improving teacher representations by imposing smoothness constraints for knowledge distillation

  • Apr 09, 2024
  • Applied Soft Computing
  • Md Imtiaz Hossain +3
  • Conference Article
  • Citations3

The Optimization Method of Knowledge Distillation Based on Model Pruning

  • Nov 06, 2020
  • Min Wu +3
  • PDF
  • Research Article
  • Citations57

A survey of model compression strategies for object detection

  • Nov 02, 2023
  • Multimedia Tools and Applications
  • Zonglei Lyu +8
  • Conference Article
  • Citations5

Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

  • May 13, 2024
  • Liyang He +6
  • Research Article
  • Citations6

Discretization and decoupled knowledge distillation for arbitrary oriented object detection

  • Apr 17, 2024
  • Digital Signal Processing
  • Cheng Chen +2
  • PDF
  • Conference Article
  • Citations70

Annealing Knowledge Distillation

  • Jan 01, 2021
  • Aref Jafari +3
  • Research Article
  • Citations8

Dual Knowledge Distillation for neural machine translation

  • Nov 09, 2023
  • Computer Speech & Language
  • Yuxian Wan +4
  • Research Article
  • Citations1

A Linguistically Interpretable Fuzzy Fault Diagnosis Model: Knowledge Distillation Perspective

  • Jan 01, 2026
  • IEEE Transactions on Fuzzy Systems
  • Meng-Wei Li +3
  • PDF
  • Research Article
  • Citations20

Collaborative Consistent Knowledge Distillation Framework for Remote Sensing Image Scene Classification Network

  • Oct 17, 2022
  • Remote Sensing
  • Shiyi Xing +4
  • Research Article
  • Citations5

Robust knowledge distillation based on feature variance against backdoored teacher model

  • Jun 24, 2024
  • Applied Soft Computing
  • Jinyin Chen +5
  • Conference Article
  • Citations3

FedKDShap: Enhancing Federated Learning via Shapley Values Driven Knowledge Distillation on Non-IID Data

  • May 08, 2025
  • Nazmus Shakib Shadin +1
  • Research Article
  • Citations6

Image quality assessment based on self-supervised learning and knowledge distillation

  • Nov 24, 2022
  • Journal of Visual Communication and Image Representation
  • Qingbing Sang +4
  • PDF
  • Research Article
  • Citations38

Knowledge Distillation in Acoustic Scene Classification

  • Jan 01, 2020
  • IEEE Access
  • Jee-Weon Jung +3
  • Research Article
  • Citations25

Quantization Robust Pruning With Knowledge Distillation

  • Jan 01, 2023
  • IEEE Access
  • Jangho Kim
  • PDF
  • Research Article
  • Citations23

A Novel Knowledge Distillation Method for Self-Supervised Hyperspectral Image Classification

  • Sep 10, 2022
  • Remote Sensing
  • Qiang Chi +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.