- https://doi.org/10.1109/actce66599.2025.00129
Construction of General Representation Model and Cross-domain Migration Method Based on Multi-modal Comparative Learning
- Sep 27, 2025
- Qianhui Yu +6 more
In order to solve the core problems of modal heterogeneity, semantic gap and model performance degradation in multi-modal data applications, this paper proposes a framework of Dynamic Domain-aware Multimodal Contrastive Learning (DD MCL), aiming at constructing a general multi-modal representation model and realizing efficient cross-domain migration. The framework is optimized through three stages: pre-training, self-adaptation and joint optimization. In the pre-training stage, Modality Attention Weight Network (MAWN) is introduced to dynamically fuse modal features, and the modes are initially aligned with standard comparison loss. In the adaptive phase, Domain-Aware Contrastive Loss (DACL) is activated for unlabeled target domain data, balancing the contrastive learning intensity between the source and target domains through dynamic weight coefficients to alleviate distribution differences between domains; In the joint optimization stage, a Memory Replay Mechanism (MRM) is introduced to construct a source domain sample "memory bank" to avoid catastrophic forgetting of the model when adapting to the target domain. At the same time, the weighted sum of source domain contrastive loss, target domain contrastive loss, and memory replay loss is used as the overall loss function to optimize the model. In the experiment, MS-COCO is used as the source domain data set, cartoon illustration data set CIBR and medical image report data set MIMIC-CXR are used as the target domain data set, ViT-B/16 and RoBERTa-base are used as encoders, and Recall@1, Recall@5 and Recall@10 of cross-modal retrieval tasks are used as evaluation indexes. The results show that the Recall@1 index of image-to-text (I2T) and text-to-image (T2I) retrieval tasks of DDMCL on CIBR and MIMIC-CXR reaches 45.7 and 43.9, respectively, which is significantly better than the baseline models such as CLIP and ALBEF. Ablation experiments verify that the core components of MAWN, DACL and MRM contribute significantly to the performance, and the model performance is the best when the domain sensing weight β is in the range of 0.6-0.8. The research shows that DDMCL can effectively realize cross-modal semantic fusion and efficient cross-domain migration without target domain labeling, and improve the generalization ability and application efficiency of the model in complex and data-scarce scenarios.
- # Source Domain
- # Target Domain
- # Target Domain Data Set
- # Source Domain Data Set
- # Memory Bank
- # Target Domain Data
- # Domain Data Set
- # Memory Replay
- # Pre-training Stage
- # Multi-modal Learning