• Home
  • Search
  • Effective Online Knowledge Distillation via Attention-Based Model Ensembling
  • Cite Icon5
  • https://doi.org/10.3390/math10224285Copy DOI Icon

Effective Online Knowledge Distillation via Attention-Based Model Ensembling

Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Large-scale deep learning models have achieved impressive results on a variety of tasks; however, their deployment on edge or mobile devices is still a challenge due to the limited available memory and computational capability. Knowledge distillation is an effective model compression technique, which can boost the performance of a lightweight student network by transferring the knowledge from a more complex model or an ensemble of models. Due to its reduced size, this lightweight model is more suitable for deployment on edge devices. In this paper, we introduce an online knowledge distillation framework, which relies on an original attention mechanism to effectively combine the predictions of a cohort of lightweight (student) networks into a powerful ensemble, and use this as a distillation signal. The proposed aggregation strategy uses the predictions of the individual students as well as ground truth data to determine a set of weights needed for ensembling these predictions. This mechanism is solely used during system training. When testing or at inference time, a single, lightweight student is extracted and used. The extensive experiments we performed on several image classification benchmarks, both by training models from scratch (on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets) and using transfer learning (on Oxford Pets and Oxford Flowers datasets), showed that the proposed framework always leads to an improvement in the accuracy of knowledge-distilled students and demonstrates the effectiveness of the proposed solution. Moreover, in the case of ResNet architecture, we observed that the knowledge-distilled model achieves a higher accuracy than a deeper, individually trained ResNet model.

Loading PDF

Similar Papers
  • Research Article
  • Citations17

Lightweight Deep Learning Models For Edge Devices—A Survey

  • Jan 06, 2025
  • International Journal of Computer Information Systems and Industrial Management Applications
  • Aminu Musa +5
  • Research Article
  • Citations8

Lightweight Deep Learning Models For Edge Devices—A Survey

  • Jan 06, 2025
  • International Journal of Computer Information Systems and Industrial Management Applications
  • Aminu Musa +5
  • Research Article
  • Citations69

MobiHisNet: A Lightweight CNN in Mobile Edge Computing for Histopathological Image Classification

  • Dec 15, 2021
  • IEEE Internet of Things Journal
  • Abhinav Kumar +5
  • Research Article
  • Citations28

A light-weight skeleton human action recognition model with knowledge distillation for edge intelligent surveillance applications

  • Dec 20, 2023
  • Applied Soft Computing
  • Cheng Dai +3
  • PDF
  • Research Article
  • Citations44

Real-Time Weed Control Application Using a Jetson Nano Edge Device and a Spray Mechanism

  • Aug 26, 2022
  • Remote Sensing
  • Eduardo Assunção +6
  • Research Article
  • Citations1

Comparative Study of CNN Architectures for Real-Time Audio-Based Car Accident Detection on Edge Devices

  • May 31, 2025
  • JOIV : International Journal on Informatics Visualization
  • Ahmada Haiz Zakiyil Ilahi +2
  • PDF
  • Research Article
  • Citations24

A Lightweight Model of Underwater Object Detection Based on YOLOv8n for an Edge Computing Platform

  • Apr 23, 2024
  • Journal of Marine Science and Engineering
  • Yibing Fan +2
  • Research Article

Detecting road damage utilizing retinanet and mobilenet models on edge devices

  • Apr 01, 2025
  • IAES International Journal of Artificial Intelligence (IJ-AI)
  • Haniah Mahmudah +3
  • Conference Article
  • Citations64

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

  • Aug 30, 2021
  • Jangho Kim +2
  • Research Article
  • Citations15

Federated learning by employing knowledge distillation on edge devices with limited hardware resources

  • Feb 14, 2023
  • Neurocomputing
  • Ehsan Tanghatari +3
  • Research Article
  • Citations2

GPT-4off: On-Board Traversability Probability Estimation for Off-Road Driving via GPT Knowledge Distillation

  • Feb 17, 2025
  • Applied Sciences
  • Nahyeong Kim +5
  • PDF
  • Research Article
  • Citations2

Designing a Deep Q-Learning Model with Edge-Level Training for Multi-Level Task Offloading in Edge Computing Networks

  • Oct 21, 2022
  • Applied Sciences
  • Ahmad Zendebudi +1
  • Conference Article
  • Citations2

AsyncFedKD: Asynchronous Federated Learning with Knowledge Distillation

  • Jun 21, 2023
  • Malik Naik Mohammed +3
  • Conference Article
  • Citations1

Split Computing based Quantized PoseNet Model for Distributed AI Architecture in IoT-Edge

  • Jan 03, 2023
  • Jyotirmoy Karjee +5
  • PDF
  • Research Article
  • Citations28

Anatomy of Deep Learning Image Classification and Object Detection on Commercial Edge Devices: A Case Study on Face Mask Detection

  • Jan 01, 2022
  • IEEE Access
  • Dimitrios Kolosov +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.