• Cite Icon1
  • https://doi.org/10.2352/ei.2024.36.8.image-238Copy DOI Icon

Adapt to Distill or Distill to Adapt

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Domain Adaptation (DA) techniques aim to overcome the domain shift between a source domain used for training and a target domain used for testing. In recent years, vision transformers have emerged as a preferred alternative to Convolutional Neural Networks (CNNs) for various computer vision tasks. When used as backbones for DA, these attention-based architectures have been found to be more powerful than standard ResNet backbones. However, vision transformers require a larger computational overhead due to their model size. In this paper, we demonstrate the superiority of attention-based architectures for domain generalization and source-free unsupervised domain adaptation. We further improve the performance of ResNet-based unsupervised DA models using knowledge distillation from a larger teacher model to the student ResNet model. We explore the efficacy of two frameworks and answer the question: is it better to distill and then adapt or to adapt and then distill? Our experiments on two popular datasets show that adapt-to-distill is the preferred approach.

Similar Papers
  • Abstract

Unsupervised Domain Adaptation of Auto-Segmentation on Multi-Source MRIs

  • Sep 29, 2023
  • International Journal of Radiation Oncology*Biology*Physics
  • W Zhang +4
  • Research Article
  • Citations37

Unsupervised Adversarial Domain Adaptation Network for Semantic Segmentation

  • Dec 26, 2019
  • IEEE Geoscience and Remote Sensing Letters
  • Wei Liu +1
  • Research Article
  • Citations17

DASRSNet: Multitask Domain Adaptation for Super-Resolution-Aided Semantic Segmentation of Remote Sensing Images

  • Jan 01, 2023
  • IEEE Transactions on Geoscience and Remote Sensing
  • Yuxiang Cai +4
  • Research Article

Improving Unsupervised Domain Adaptation via Multiple Adversarial Learning

  • Oct 01, 2023
  • 電腦學刊
  • Yu-Dong Cao Yu-Dong Cao +2
  • PDF
  • Research Article
  • Citations1

Correlation enhanced distribution adaptation for prediction of fall risk

  • Feb 12, 2024
  • Scientific Reports
  • Ziqi Guo +4
  • Research Article
  • Citations31

Improving the Transferability of Deep Learning Models for Crop Yield Prediction: A Partial Domain Adaptation Approach

  • Sep 16, 2023
  • Remote Sensing
  • Yuchi Ma +3
  • Research Article
  • Citations7

Unsupervised Domain Expansion for Visual Categorization

  • Nov 12, 2021
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Jie Wang +4
  • Research Article
  • Citations3

Robust Deep Softmax Regression Against Label Noise for Unsupervised Domain Adaptation

  • Jun 07, 2019
  • International Journal of Pattern Recognition and Artificial Intelligence
  • Guangbin Wu +4
  • Conference Article
  • Citations80

Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation

  • Jan 01, 2021
  • L.T Nguyen-Meidine +5
  • Book Chapter
  • Citations8

Domain Adaptation for Eye Segmentation

  • Jan 01, 2020
  • Yiru Shen +2
  • Conference Article
  • Citations8

Learning Target Predictive Function without Target Labels

  • Dec 01, 2012
  • Chun-Wei Seah +3
  • PDF
  • Research Article
  • Citations5

On the Importance of Attention and Augmentations for Hypothesis Transfer in Domain Adaptation and Generalization

  • Oct 12, 2023
  • Sensors (Basel, Switzerland)
  • Rajat Sahay +5
  • Research Article
  • Citations24

Semi-supervised domain adaptation for segmentation models on different monitoring settings

  • May 01, 2023
  • Automation in Construction
  • Yeji Hong +4
  • Conference Article
  • Citations72

Exploring Category-Agnostic Clusters for Open-Set Domain Adaptation

  • Jun 01, 2020
  • Yingwei Pan +4
  • Preprint Article

Segmentation Cycle-GAN Loss for Structure-Preserving Domain Adaptation of Medical Images

  • Sep 06, 2024
  • Paolo Iacono
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.