A Study on improving Vision Transformer performance based on object frequency characteristic classification
In recent deep learning-based image classification studies, Vision Transformer has shown high performance and has the potential to replace CNN. In this study, we propose a new structure that integrates frequency domain information based on Fourier Transform into the ViT structure. The proposed model effectively combines spatial-frequency information of images while maintaining the global representation learning ability of the self-attention mechanism, thereby enabling more sophisticated visual feature representation. The training and validation loss curves showed that the proposed model showed an overall stable learning process, had little overfitting, and converged to the lowest validation loss. In the class-by-class accuracy evaluation, the highest accuracy was recorded in the Automobile, Deer, Horse, and Ship classes, and in particular, the Ship class showed a significant performance improvement compared to other models. This suggests that frequency-based information is effective for recognizing objects with repetitive structures or distinct outlines. On the other hand, in the Dog and Cat classes with complex backgrounds or ambiguous shapes, it showed a slight difference, but overall, it maintained a balanced performance. the proposed model showed superior performance in learning stability and classification accuracy compared to the existing ViT, and showed a particularly strong effect in tasks that require simultaneous consideration of object shape information and periodic patterns. Proposed frequency-based Attention mechanism suggests the possibility of expanding the Transformer series vision model, and is expected to be effectively utilized in various visual recognition problems such as medical imaging, satellite imaging, autonomous driving, and multimodal learning. In future studies, it can be expanded into a model applicable to real-time environments through fusion with various frequency transformation techniques and application of lightweight structures.
Read more