• Home
  • Search
  • PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
  • Cite Icon64
  • https://doi.org/10.21437/interspeech.2021-248Copy DOI Icon

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

  • Aug 30, 2021
  • Jangho Kim +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

As edge devices become prevalent, deploying Deep Neural Networks (DNN) on edge devices has become a critical issue. However, DNN requires a high computational resource which is rarely available for edge devices. To handle this, we propose a novel model compression method for the devices with limited computational resources, called PQK consisting of pruning, quantization, and knowledge distillation (KD) processes. Unlike traditional pruning and KD, PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. PQK has two phases. Phase 1 exploits iterative pruning and quantization-aware training to make a lightweight and power-efficient model. In phase 2, we make a teacher network by adding unimportant weights unused in phase 1 to a pruned network. By using this teacher network, we train the pruned network as a student network. In doing so, we do not need a pre-trained teacher network for the KD framework because the teacher and the student networks coexist within the same network. We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.

Similar Papers
  • Research Article

An Ordered Aggregation-Based Ensemble Selection Method of Lightweight Deep Neural Networks With Random Initialization

  • Jan 01, 2022
  • IEEE Access
  • Lin He +2
  • Research Article

Compressing Neural Networks on Limited Computing Resources

  • Jan 01, 2025
  • IEEE Access
  • Seunghyun Lee +4
  • Dissertation

Deep learning acceleration on edge devices with algorithm/hardware co-design

  • Jan 01, 2022
  • Mengshu Sun
  • Conference Article
  • Citations3

Learning From Music to Visual Storytelling of Shots: A Deep Interactive Learning Mechanism

  • Oct 12, 2020
  • Jen-Chun Lin +4
  • Research Article
  • Citations15

Federated learning by employing knowledge distillation on edge devices with limited hardware resources

  • Feb 14, 2023
  • Neurocomputing
  • Ehsan Tanghatari +3
  • Research Article
  • Citations11

Efficient face anti-spoofing via head-aware transformer based knowledge distillation with 5 MB model parameters

  • Sep 10, 2024
  • Applied Soft Computing
  • Jun Zhang +6
  • Research Article
  • Citations17

Lightweight Deep Learning Models For Edge Devices—A Survey

  • Jan 06, 2025
  • International Journal of Computer Information Systems and Industrial Management Applications
  • Aminu Musa +5
  • Research Article
  • Citations8

Lightweight Deep Learning Models For Edge Devices—A Survey

  • Jan 06, 2025
  • International Journal of Computer Information Systems and Industrial Management Applications
  • Aminu Musa +5
  • Conference Article
  • Citations3

Plant Leaf Segmentation Using Knowledge Distillation

  • Oct 26, 2022
  • Joo-Yeon Jung +2
  • Dissertation
  • Citations1

Towards energy-efficient deep learning for sustainable AI

  • Jan 01, 2024
  • Yifan Gong
  • Conference Article
  • Citations5

An Intelligent Bearing Fault Diagnosis based on Modified Probabilistic Knowledge Distillation

  • Oct 15, 2021
  • 2021 Global Reliability and Prognostics and Health Management (PHM-Nanjing)
  • Ziqian Shen +1
  • Research Article
  • Citations15

Dual discriminator adversarial distillation for data-free model compression

  • Oct 25, 2021
  • International Journal of Machine Learning and Cybernetics
  • Haoran Zhao +5
  • Research Article
  • Citations38

Data-free knowledge distillation in neural networks for regression

  • Mar 06, 2021
  • Expert Systems with Applications
  • Myeonginn Kang +1
  • Research Article
  • Citations12

Adversarial learning-based multi-level dense-transmission knowledge distillation for AP-ROP detection.

  • Feb 01, 2023
  • Medical Image Analysis
  • Hai Xie +8
  • Research Article
  • Citations6

Image quality assessment based on self-supervised learning and knowledge distillation

  • Nov 24, 2022
  • Journal of Visual Communication and Image Representation
  • Qingbing Sang +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.