• Home
  • Search
  • Self-distilled Vision Transformer for Domain Generalization
  • Open Access IconOpen Access
  • Cite Icon31
  • https://doi.org/10.1007/978-3-031-26284-5_17Copy DOI Icon

Self-distilled Vision Transformer for Domain Generalization

  • Jan 1, 2023
  • Maryam Sultana +4 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Abstract In the recent past, several domain generalization (DG) methods have been proposed, showing encouraging performance, however, almost all of them build on convolutional neural networks (CNNs). There is little to no progress on studying the DG performance of vision transformers (ViTs), which are challenging the supremacy of CNNs on standard benchmarks, often built on i.i.d assumption. This renders the real-world deployment of ViTs doubtful. In this paper, we attempt to explore ViTs towards addressing the DG problem. Similar to CNNs, ViTs also struggle in out-of-distribution scenarios and the main culprit is overfitting to source domains. Inspired by the modular architecture of ViTs, we propose a simple DG approach for ViTs, coined as self-distillation for ViTs. It reduces the overfitting of source domains by easing the learning of input-output mapping problem through curating non-zero entropy supervisory signals for intermediate transformer blocks. Further, it does not introduce any new parameters and can be seamlessly plugged into the modular composition of different ViTs. We empirically demonstrate notable performance gains with different DG baselines and various ViT backbones in five challenging datasets. Moreover, we report favorable performance against recent state-of-the-art DG methods. Our code along with pre-trained models are publicly available at: https://github.com/maryam089/SDViT.KeywordsDomain generalizationVision transformersSelf distillation

Similar Papers
  • Research Article
  • Citations25

Domain Generalization for Face Anti-Spoofing via Negative Data Augmentation

  • Jan 01, 2023
  • IEEE Transactions on Information Forensics and Security
  • Weihang Wang +4
  • Dissertation
  • Citations1

Domain adaptation and generalization for visual recognition

  • Jan 01, 2017
  • Li Niu
  • Research Article
  • Citations2

Multisource Domain Separation Network for Industrial Intelligent Monitoring in Unseen Conditions.

  • Jan 01, 2025
  • IEEE transactions on cybernetics
  • Zidi Jia +3
  • Research Article
  • Citations13

PLACE Dropout: A Progressive Layer-wise and Channel-wise Dropout for Domain Generalization

  • Oct 23, 2023
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Jintao Guo +3
  • Conference Article
  • Citations9

Diverse Intra- and Inter-Domain Activity Style Fusion for Cross-Person Generalization in Activity Recognition

  • Aug 24, 2024
  • Junru Zhang +6
  • Research Article
  • Citations11

Learning Category-Invariant Disentangled Features for Domain Generalization in Machine Fault Diagnosis

  • Jan 01, 2026
  • IEEE/ASME Transactions on Mechatronics
  • Zuoyi Chen +3
  • Research Article
  • Citations1

DI2SDiff++: Activity Style Decomposition and Diffusion-Based Fusion for Cross-Person Generalization in Activity Recognition

  • Jan 01, 2025
  • IEEE Transactions on Mobile Computing
  • Junru Zhang +6
  • Conference Article
  • Citations51

Federated Domain Generalization for Image Recognition via Cross-Client Style Transfer

  • Jan 01, 2023
  • Junming Chen +3
  • Research Article
  • Citations15

Fault Diagnosis of Unseen Modes in Chemical Processes Based on Labeling and Class Progressive Adversarial Learning

  • Jan 01, 2023
  • IEEE Transactions on Instrumentation and Measurement
  • Yutang Xiao +5
  • Book Chapter
  • Citations394

Learning to Generate Novel Domains for Domain Generalization

  • Jan 01, 2020
  • Kaiyang Zhou +3
  • Research Article
  • Citations1

An adversarial-based domain generalization method for the health evaluation of axial piston pumps

  • Sep 04, 2024
  • Physica Scripta
  • Yuechen Shao +3
  • Conference Article
  • Citations5

Simple Domain Generalization Methods are Strong Baselines for Open Domain Generalization

  • Jun 30, 2024
  • Masashi Noguchi +1
  • Conference Article
  • Citations1436

Deeper, Broader and Artier Domain Generalization

  • Oct 01, 2017
  • Da Li +3
  • Research Article
  • Citations23

Generative Inference Network for Imbalanced Domain Generalization.

  • Jan 01, 2023
  • IEEE Transactions on Image Processing
  • Haifeng Xia +2
  • Book Chapter
  • Citations7

Generalizing Nucleus Recognition Model in Multi-source Ki67 Immunohistochemistry Stained Images via Domain-Specific Pruning

  • Jan 01, 2021
  • Jiatong Cai +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.