Learning From Expert: Vision-Language Knowledge Distillation for Unsupervised Cross-Modal Hashing Retrieval

In order to further improve the performance of cross-modal retrieval, a cross-modal hash retrieval method integrating multi-level similarity information is proposed. First, self-attention method is used to enhance the text features, and a new fusion feature is constructed based on the original features and hash features of different modalities. Then, based on these three features, three auxiliary similarity matrices are constructed, and the fourth auxiliary similarity matrix is constructed by a weighted combination method. Finally, these four different matrices are used to calculate the loss functions between different similarity matrices and between different modalities. Since the four matrices include different feature forms and different matrix construction methods, they can better express similarity information of different modalities and improve the retrieval performance. The experiments are conducted on three benchmark datasets of Wikipedia, MIRFlickr and NUS-WIDE. The results show that the mAP values at different code bits of proposed method is better than that of many state-of-the-art methods, which verifies the effectiveness and robustness of our method.

Cross-modal hashing (CMH) maps heterogeneous multiple modality data into compact binary code to achieve fast and flexible retrieval across different modalities, especially in large-scale retrieval. As the data don't need a lot of manual annotation, unsupervised cross-modal hashing has a wider application prospect than supervised method. However, the existing unsupervised methods are difficult to achieve satisfactory performance due to the lack of credible supervisory information. To solve this problem, inspired by knowledge distillation, we propose a novel unsupervised Knowledge Distillation Cross-Modal Hashing method (KDCMH), which can use similarity information distilled from unsupervised method to guide supervised method. Specifically, firstly, the teacher model adopted an unsupervised distribution-based similarity hashing method, which can construct a modal fusion similarity matrix.Secondly, under the supervision of teacher model distillation information, student model can generate more discriminative hash codes. In two public datasets NUS-WIDE and MIRFLICKR-25K, extensive experiments have proved the significant improvement of KDCMH on several representative unsupervised cross-modal hashing methods.

Objective. Over the past years, convolutional neural networks based methods have dominated the field of medical image segmentation. But the main drawback of these methods is that they have difficulty representing long-range dependencies. Recently, the Transformer has demonstrated super performance in computer vision and has also been successfully applied to medical image segmentation because of the self-attention mechanism and long-range dependencies encoding on images. To the best of our knowledge, only a few works focus on cross-modalities of image segmentation using the Transformer. Hence, the main objective of this study was to design, propose and validate a deep learning method to extend the application of Transformer to multi-modality medical image segmentation. Approach. This paper proposes a novel automated multi-modal Transformer network termed AMTNet for 3D medical image segmentation. Especially, the network is a well-modeled U-shaped network architecture where many effective and significant changes have been made in the feature encoding, fusion, and decoding parts. The encoding part comprises 3D embedding, 3D multi-modal Transformer, and 3D Co-learn down-sampling blocks. Symmetrically, the 3D Transformer block, upsampling block, and 3D-expanding blocks are included in the decoding part. In addition, a Transformer-based adaptive channel interleaved Transformer feature fusion module is designed to fully fuse features of different modalities. Main results. We provide a comprehensive experimental analysis of the Prostate and BraTS2021 datasets. The results show that our method achieves an average DSC of 0.907 and 0.851 (0.734 for ET, 0.895 for TC, and 0.924 for WT) on these two datasets, respectively. These values show that AMTNet yielded significant improvements over the state-of-the-art segmentation networks. Significance. The proposed 3D segmentation network exploits complementary features of different modalities during the feature extraction process at multiple scales to increase the 3D feature representations and improve the segmentation efficiency. This powerful network enriches the research of the Transformer to multi-modal medical image segmentation.

Existing unsupervised cross-modal hashing retrieval methods generally are restricted by two limitations: 1. Most existing cross-modal hashing methods explore modal semantic information based on paired data, while the semantic information available in unpaired single-modal data is discarded; unsuper-vised methods that rely only on constructing semantic matrix for retrieval have considerable limitations, and the similarity information between heterogeneous data requires deep mining. In this paper, we propose an unsupervised hash retrieval based on colla-borative semantic distribution (UPJS). First, the method constr-ucts a collaborative semantic similarity matrix between modali-ties, which fuses the original semantic information of heterogen-eous modalities and preserves the semantic connections between different modal instances. Secondly, this paper employs feature fusion to transform unpaired information into paired information, and then achieves semantic similarity by considering both paired and unpaired information. Furthermore, the method constructs inter-modal triplet constraints between paired and unpa-ired data, and utilizes a collaborative matrix to jointly train se-mantic binary codes. Extensive experiments on two benchmark datasets, MIRFlickr and WikiData, show that this method has a favorable performance in both paired and unpaired data.

Nowadays cross-media retrieval is an useful technology that helps people find expected information from the huge amount of multimodal data more efficiently. A common cross-media retrieval framework is first to map features of different modalities into an isomorphic semantic space so that the similarity between heterogeneous data can be measured. For most of semantic space based methods, the mapping mechanism from original to semantic space of each modality is optimized independently, yet the more discriminative characteristic of a certain modality is not taken into account. In this paper, we propose a deep framework which introduces a latent embedding layer to learn joint parameters to obtain semantically meaningful representations of images and texts. Specifically, the discriminative characteristic embedded in the textual modality can be transferred to images through the latent embedding layer and joint parameters to enhance the consistency between semantic representations. Extensive experiments on the three popular publicly available datasets well demonstrate the superiority of the proposed method, which achieves the new state-of-the-arts.

Deep learning is skilled at learning representation from raw data, which are embedded in the semantic space. Traditional multimodal networks take advantage of this, and maximize the joint distribution over the representations of different modalities. However, the similarity among the representations are not emphasized, which is an important property for multimodal data. In this paper, we will introduce a novel learning method for multimodal networks, named as Semantic Similarity Learning (SSL), which aims at training the model via enhancing the similarity between the high-level features of different modalities. Sets of experiments are conducted for evaluating the method on different multimodal networks and multiple tasks. The experimental results demonstrate the effectiveness of SSL in keeping the shared information and improving the discrimination. Particularly, SSL shows its ability in encouraging each modality to learn transferred knowledge from the other one when faced with missing data.

For patients with brain tumors, effectively utilizing the complementary information between multimodal medical images is crucial for accurate lesion segmentation. However, effectively utilizing the complementary features across different modalities remains a challenging task. To address these challenges, we propose a modal feature supplement network (MFSNet), which extracts modality features simultaneously using both a main and an auxiliary network. During this process, the auxiliary network supplements the modality features of the main network, enabling accurate brain tumor segmentation. We also design a modal feature enhancement module (MFEM), a cross‐layer feature fusion module (CFFM), and an edge feature supplement module (EFSM). MFEM enhances the network performance by fusing the modality features from the main and auxiliary networks. CFFM supplements additional contextual information by fusing features from adjacent encoding layers at different scales, which are then passed into the corresponding decoding layers. This aids the network in preserving more details during upsampling. EFSM improves network performance by using deformable convolution to extract challenging boundary lesion features, which are then used to supplement the final output of the decoding layer. We evaluated MFSNet on the BraTS2018 and BraTS2021 datasets. The Dice scores for the whole tumor, tumor core, and enhancing tumor regions were 90.86%, 90.59%, 84.72%, and 92.28%, 92.47%, 86.07%, respectively. This validates the accuracy of MFSNet in brain tumor segmentation, demonstrating its superiority over other networks of similar type.

We propose a novel method based on teacher-student learning framework for 3D human pose estimation without any 3D annotation or side information. To solve this unsupervised-learning problem, the teacher network adopts pose-dictionary-based modeling for regularization to estimate a physically plausible 3D pose. To handle the decomposition ambiguity in the teacher network, we propose a cycle-consistent architecture promoting a 3D rotation-invariant property to train the teacher network. To further improve the estimation accuracy, the student network adopts a novel graph convolution network for flexibility to directly estimate the 3D coordinates. Another cycle-consistent architecture promoting 3D rotation-equivariant property is adopted to exploit geometry consistency, together with knowledge distillation from the teacher network to improve the pose estimation performance. We conduct extensive experiments on Human3.6M and MPI-INF-3DHP. Our method reduces the 3D joint prediction error by 11.4% compared to state-of-the-art unsupervised methods and also outperforms many weakly-supervised methods that use side information on Human3.6M. Code will be available at https://github.com/sjtuxcx/ITES.

With the rise of e-commerce, personalized recommendation algorithms have received much attention in recent years. Meanwhile, multimodal recommendation algorithms have become the next competitive track of personalized recommendation. The current mainstream recommendation algorithms (e.g., NeuMF, NGCF) rely too much on empirical data and do not effectively utilize multimodal information to model user preferences, while there is the problem of sparsity of user preference data. Secondly, they are still limited to using the same weights to fuse the item features of different modalities, and have the problem of inaccurate recommendation. For this reason, this paper proposes Multi-Modal Self-Supervised Learning Algorithm Based Product Recommendation (MMSLRec). (1) Data augmentation by self-supervised adversarial perturbation generation learning is performed to supplement the missing interaction information of tailed products to solve the common data sparsity problem in product recommendation. (2) A multimodal representation learning module is introduced to capture deep features in image and text modalities using AlexNet and Bert, respectively, in order to avoid the recommendation accuracy degradation problem caused by insufficient data feature learning. (3) Learning the user's perceived strength in different modalities, using the user co-occurrence matrix to fuse the features and make recommendations. Experiments on Amazon-Electronics, Amazon-Clothing, and Amazon-Baby, with different degrees of data sparsity verify the effectiveness of our MMSLRec in solving the data sparsity problem and the multimodal user preference modeling problem, and compared to the state-of-the-art baseline recommendation model on the three datasets, the Recall@20 improved by 9.26%, 3.11% and 2.92%, respectively.

Deep learning-based detection of depression by fusing auditory, visual and textual clues.

RGB-D cross-modal person re-identification (Re-ID) task aims to match the person images between the RGB and depth modalities. This task is rather challenging for the tremendous discrepancy between these two modalities in addition to common issues such as lighting conditions, human posture, camera angle, etc. Nowadays only few types of research focus on this task, and existing Re-ID methods tend to learn homogeneous structural relationships in an image, which have limited discriminability and weak robustness to noisy images. In this paper, we propose A Local-Global Interaction Network dedicated to processing cross-modal problems. The network can constrain the center distance between two modals, and improve the intra-class cross-modality similarity. Besides, it can also learn the local and global features of different modalities to enrich the features extracted from different modes. We validate the effectiveness of our approach on public benchmark datasets. Experimental results demonstrate our method outperforms other state-of-the-arts in terms of visual quality and quantitative measurement.KeywordsCross-modal person re-identificationGlobal self-attentionIntra-class cross-modality similarity

A radiomics model for preoperative prediction of brain invasion in meningioma non-invasively based on MRI: A multicentre study

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology.

Multi-antenna intelligent spectrum sensing in the presence of non-Gaussian interference

A Radiomics Model for Preoperative Prediction of Brain Invasion in Meningioma Non-Invasively Based on MRI: A Multicentre Study