Cross-Media Retrieval via Deep Semantic Canonical Correlation Analysis and Logistic Regression

Cross-media retrieval is becoming increasingly important nowadays. To address this challenging problem, most existing approaches project heterogeneous features into a unified feature space to facilitate their similarity computation. However, this unified feature space usually has no explicit semantic meanings, which might ignore the hints contained in the original media content, and thus is not able to fully measure the similarities among different media types. By considering the above issues, we propose a new approach to cross-media retrieval via semantic entity projection SEP in this paper. Our approach consists of three main steps. Firstly, an entity level with fine-grained semantics between low-level features and high-level concepts are constructed, so as to help bridge the semantic gap to a certain extent. Then, an entity projection is learned by minimizing both cross-media correlation error and single-media reconstruction error from low-level features to the entity level, with which a unified feature space with explicit semantic meanings can be obtained from low-level features. Finally, the semantic abstraction of high-level concepts is generated by using logistic regression to conduct cross-media retrieval. Experimental results on the Wikipedia dataset show the effectiveness of the proposed approach.

Takashima, Naoki Li, Frédéric Grzegorzek, Marcin Shirahama, KimiakiMusic-emotion retrieval is important for treatments of mood disorders and depression based on music. Although existing approaches only investigate one-way retrieval from music to emotions, an approach from emotion to music might be needed for proper application of music-based therapies. This can be achieved by sensor-based music retrieval which firstly recognises a specific emotion based on physiological data acquired by wearable sensors and then identifies music suitable for that emotion. In this paper, we propose Cross-modal Music-emotion Retrieval (CMR) as the first step to achieve retrieval of music based on sensor data. Our approach uses Deep Canonical Correlation Analysis (DeepCCA) which projects music samples and their associated emotion sequences into a common space using deep neural networks, and maximises the correlation between projected music samples and emotion sequences using CCA. Our experiments show the superiority of our approach for CMR compared to one-way retrieval.

Scene change detection is the process of identifying the differences between the multi-temporal image scenes at the semantic level, which has significant potential in the application of urban development and land management. In this paper, we propose a novel deep convolution canonical correlation analysis neural network (DCCANet) architecture, which could consider the spectral-spatial-temporal correlation feature for scene change detection in remote sensing images. For this purpose, we put together the convolutional neural networks (CNNs) and the deep canonical correlation analysis (DCCA) into the end-to-end network. The CNN could get the spectral-spatial feature information for scene representation, while the later could enhance the temporal correlation by nonlinear high- dimensional transformations between the multi-temporal image scenes for scene change detection. Experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in scene classification and change detection.

Cross-media retrieval of scientific and technological information based on multi-feature fusion

For cross-modal subspace clustering, the key point is how to exploit the correlation information between cross-modal data. However, most hierarchical and structural correlation information among cross-modal data cannot be well exploited due to its high-dimensional non-linear property. To tackle this problem, in this paper, we propose an unsupervised framework named Cross-Modal Subspace Clustering via Deep Canonical Correlation Analysis (CMSC-DCCA), which incorporates the correlation constraint with a self-expressive layer to make full use of information among the inter-modal data and the intra-modal data. More specifically, the proposed model consists of three components: 1) deep canonical correlation analysis (Deep CCA) model; 2) self-expressive layer; 3) Deep CCA decoders. The Deep CCA model consists of convolutional encoders and correlation constraint. Convolutional encoders are used to obtain the latent representations of cross-modal data, while adding the correlation constraint for the latent representations can make full use of the information of the inter-modal data. Furthermore, self-expressive layer works on latent representations and constrain it perform self-expression properties, which makes the shared coefficient matrix could capture the hierarchical intra-modal correlations of each modality. Then Deep CCA decoders reconstruct data to ensure that the encoded features can preserve the structure of the original data. Experimental results on several real-world datasets demonstrate the proposed method outperforms the state-of-the-art methods.

Existing cross-media retrieval methods are usually conducted under the supervised setting, which need lots of annotated training data. Generally, it is extremely labor-consuming to annotate cross-media data. So unsupervised cross-media retrieval is highly demanded, which is very challenging because it has to handle heterogeneous distributions across different media types without any annotated information. To address the above challenge, this paper proposes Domain Adaptation with Scene Graph (DASG) approach, which transfers knowledge from the source domain to improve cross-media retrieval in the target domain. Our DASG approach takes Visual Genome as the source domain, which contains image knowledge in the form of scene graph. The main contributions of this paper are as follows: First, we propose to address unsupervised cross-media retrieval by domain adaptation. Instead of using the labor-consuming annotated information of cross-media data in the training stage, our DASG approach learns cross-media correlation knowledge from Visual Genome, and then transfers the knowledge to cross-media retrieval through media alignment and distribution alignment. Second, our DASG approach utilizes fine-grained information via scene graph representation to enhance generalization capability across domains. The generated scene graph representation builds (subject$\rightarrow $ relationship$\rightarrow $ object) triplets by exploiting objects and relationships within image and text, which makes the cross-media correlation more precise and promotes unsupervised cross-media retrieval. Third, we exploit the related tasks including object and relationship detection for learning more discriminative features across domains. Leveraging the semantic information of objects and relationships improves cross-media correlation learning for retrieval. Experiments on two widely-used cross-media retrieval datasets, namely Flickr-30K and MS-COCO, show the effectiveness of our DASG approach.

Deep neural networks have demonstrated their effectiveness in computer vision, especially for image classification and detection. Mixup is recently proposed as a data augmentation scheme, which applies linear combination of two random training examples and corresponding targets. However, the linear assumption is inappropriate for training a non-linear model. In this paper, we propose a self-supervised method which requires the consistency of original and mixed images on feature space. Our work is motived by the semantic information is related to the relative position of features. To implement this idea effectively and efficiently, we perform two-stage of training procedure, i.e., running estimation of class centers of original data in feature space, and training deep neural networks with modified loss term of Mixup. Besides, the proposed approach is also compatible with other variants of Mixup. We validate our approach on two popular image classification datasets, CIFAR10 and CIFAR100 by a variety of advanced deep neural networks, and demonstrate consistent generalization improvements, sometimes significantly. We also conduct analytical experiments to evaluate the robustness of our method to hyperparameters.

In complex underwater environments, the single mode of a single sensor cannot meet the precision requirement of object identification, and multisource fusion is currently the mainstream research approach. Deep canonical correlation analysis is an efficient feature fusion method but suffers from problems such as not strong scalability and low efficiency. Therefore, an improved deep canonical correlation analysis fusion method is proposed for underwater multisource sensor data containing noise. First, a denoising autoencoder is used for denoising and to reduce the data dimension to extract new feature expressions of raw data. Second, given that underwater acoustic data can be characterized as 1-dimensional time series, a 1-dimensional convolutional neural network is used to improve the deep canonical correlation analysis model, and multilayer convolution and pooling are implemented to decrease the number of parameters and increase the efficiency. To improve the scalability and robustness of the model, a stochastic decorrelation loss function is used to optimize the objective function, which reduces the algorithm complexity from O(n³) to O(n²). The comparison experiment of the proposed algorithm and other typical algorithms on MNIST containing noise and underwater multisource data in different scenes shows that the proposed algorithm is superior to others regardless of the efficiency or precision of target classification.

Batch process monitoring is of great importance to ensure the stable operation during the process running. However, traditional deep learning methods have certain limitations when dealing with complex data structures and dynamic features that are prominent in industrial batch processes. This paper proposes a GraphSAGE-LSTM-based deep canonical correlation analysis (GSA-LSTM-CCA) to enhance the performance of feature learning for batch process monitoring. First, the k-nearest neighbor method is used to construct a graph for the variable-unfolding batch process data as a supplementary structural information for the original spatiotemporal correlations. Then, GraphSAGE and LSTM are used to extract the structural features and time-varying features of data, respectively. Finally, the learned features are weighted and fused to construct a residual generator, and CCA is utilized to carry out process monitoring. The effectiveness and superiority of the proposed method are verified in an actual injection molding process.

Linkage influence of energy market on financial market by multiscale complexity synchronization

Inference of hidden common driver dynamics by anisotropic self-organizing neural networks.

This paper investigates attention fatigue detection and multi-grain emotional AI classification for student mental health through an educational brain-computer interface (BCI), with a focus on integrating biomechanical principles to enhance understanding and application. Recognizing the growing importance of students’ mental health and well-being, this study introduces a domain generalization approach in transfer learning to improve cross-subject BCI model accuracy, addressing the challenges of individual variability. The proposed model utilizes only seven electrodes and achieves a 90% accuracy rate in differentiating between two cognitive-behavioral tasks. A truncated weighting algorithm is employed to optimize electrode combinations, enabling effective generalization across subjects. To tackle the practical challenges of emotion recognition in educational settings, the study reduces data sampling points by identifying key brain regions and frequency bands associated with emotions. Machine learning algorithms, including support vector machines (SVM), Bayesian networks, and K-nearest neighbor (KNN), further enhance recognition accuracy. By integrating eye movement and electroencephalography (EEG) signals using deep canonical correlation analysis, the model achieves cumulative accuracy improvements of 15% and 12% compared to unimodal EEG and eye movement data, respectively, across 12 subjects. Incorporating biomechanical principles, the study also examines the mechanical properties of neural tissues and their influence on signal propagation. By analyzing the viscoelastic behavior of brain tissue and its impact on EEG signal transmission, the research provides insights into how mechanical stress and strain affect neural activity. This biomechanical perspective enhances the understanding of individual variability in EEG signals and contributes to the development of more robust and personalized BCI models. The integration of biomechanics with AI-driven emotion classification and attention fatigue detection offers a comprehensive approach to improving student mental health and educational outcomes. This fusion approach demonstrates superior performance in both emotion classification and attention fatigue detection, offering substantial potential for real-time interventions in student mental health and the enhancement of educational outcomes.

Conventional Speech Emotion Recognition (SER) approaches put more emphasis on extracting magnitude spectrum-based features, such as Mel Frequency Cepstral Coefficients (MFCCs), and Mel spectrogram. However, phase information is ignored due to signal processing difficulties such as the phase wrapping issue. This work develops a multichannel Convolution Neural Network-Bidirectional Long Short Term Memory (CNN-BLSTM) architectures with an attention mechanism for speaker-independent SER by considering phase and magnitude spectrum-based features. The phase-based features are extracted using the Modified Group Delay Function (MODGD). The obtained phase features are combined with MFCC features. The CNN-BLSTM network extract learned representation from magnitude and phase features. The learned representation from MFCCs and MODGD are combined and given as an input to the Support Vector Machine (SVM) for classification. The Deep Canonical Correlation Analysis (DCCA) is used to maximize the correlation between magnitude and phase information to improve the conventional SER system’s performance. Here the IEMOCAP database is used for performance analysis. The experimental results show improvement over MFCC features and existing approaches for unimodal SER. In this work, we also developed real-time Web server application for the proposed architecture.

Age-related macular degeneration (AMD) is a prevalent retinal degenerative disease among the elderly and is a major cause of irreversible vision loss worldwide. Although color fundus photography (CFP) and optical coherence tomography (OCT) are widely used for AMD diagnosis, information from a single modal is inadequate to fully capture the complex pathological features of AMD. To address this, this study proposes an innovative multi-modal deep learning framework that fine-tunes pre-trained single-modal retinal models for efficient application in multi-modal AMD categorization tasks. Specifically, two independent vision transformer models are used to extract features from CFP and OCT images, followed by deep canonical correlation analysis (DCCA) to perform nonlinear mapping and fusion of features from both modalities, maximizing cross-modal feature correlation. Moreover, to reduce the computational complexity of multi-modal integration, we introduce the low-rank adaptation (LoRA) technique, which uses low-rank decomposition of parameter matrices, achieving superior performance compared to full fine-tuning with only about 0.49% of the trainable parameters. Experimental results on the public dataset MMC-AMD validate the framework's effectiveness. The proposed model achieves an overall F1-score of 0.948, AUC-ROC of 0.991, and accuracy of 0.949, significantly outperforming existing single-modal and multi-modal baseline models, particularly excelling in recognizing complex pathological categories.

This paper tackles a fine-grained action recognition task that aims to identify pitch-type-specific habits in baseball pitching motions. This task is highly challenging because it requires distinguishing subtle differences among pitch types within the common motion of “throwing.” Although it scarcely addressed in previous studies, it holds great potential for practical applications such as sports performance analysis. We propose a novel multimodal approach that integrates skeleton and RGB modalities. First, the Segment Anything Model (SAM) is employed to remove background information from RGB videos, thereby mitigating background bias. Next, Deep Canonical Correlation Analysis (DCCA) is applied to align the skeleton and RGB features. By further decomposing the RGB features into structural and textural components, the proposed method enhances discriminative performance. Experiments conducted on a custom pitching dataset collected from three pitchers demonstrate that our method outperforms conventional approaches in classification accuracy and effectively captures subtle pitching habits.