Multi-modal Sentiment Analysis using Deep Canonical Correlation Analysis

For cross-modal subspace clustering, the key point is how to exploit the correlation information between cross-modal data. However, most hierarchical and structural correlation information among cross-modal data cannot be well exploited due to its high-dimensional non-linear property. To tackle this problem, in this paper, we propose an unsupervised framework named Cross-Modal Subspace Clustering via Deep Canonical Correlation Analysis (CMSC-DCCA), which incorporates the correlation constraint with a self-expressive layer to make full use of information among the inter-modal data and the intra-modal data. More specifically, the proposed model consists of three components: 1) deep canonical correlation analysis (Deep CCA) model; 2) self-expressive layer; 3) Deep CCA decoders. The Deep CCA model consists of convolutional encoders and correlation constraint. Convolutional encoders are used to obtain the latent representations of cross-modal data, while adding the correlation constraint for the latent representations can make full use of the information of the inter-modal data. Furthermore, self-expressive layer works on latent representations and constrain it perform self-expression properties, which makes the shared coefficient matrix could capture the hierarchical intra-modal correlations of each modality. Then Deep CCA decoders reconstruct data to ensure that the encoded features can preserve the structure of the original data. Experimental results on several real-world datasets demonstrate the proposed method outperforms the state-of-the-art methods.

Scene change detection is the process of identifying the differences between the multi-temporal image scenes at the semantic level, which has significant potential in the application of urban development and land management. In this paper, we propose a novel deep convolution canonical correlation analysis neural network (DCCANet) architecture, which could consider the spectral-spatial-temporal correlation feature for scene change detection in remote sensing images. For this purpose, we put together the convolutional neural networks (CNNs) and the deep canonical correlation analysis (DCCA) into the end-to-end network. The CNN could get the spectral-spatial feature information for scene representation, while the later could enhance the temporal correlation by nonlinear high- dimensional transformations between the multi-temporal image scenes for scene change detection. Experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in scene classification and change detection.

In complex underwater environments, the single mode of a single sensor cannot meet the precision requirement of object identification, and multisource fusion is currently the mainstream research approach. Deep canonical correlation analysis is an efficient feature fusion method but suffers from problems such as not strong scalability and low efficiency. Therefore, an improved deep canonical correlation analysis fusion method is proposed for underwater multisource sensor data containing noise. First, a denoising autoencoder is used for denoising and to reduce the data dimension to extract new feature expressions of raw data. Second, given that underwater acoustic data can be characterized as 1-dimensional time series, a 1-dimensional convolutional neural network is used to improve the deep canonical correlation analysis model, and multilayer convolution and pooling are implemented to decrease the number of parameters and increase the efficiency. To improve the scalability and robustness of the model, a stochastic decorrelation loss function is used to optimize the objective function, which reduces the algorithm complexity from O(n³) to O(n²). The comparison experiment of the proposed algorithm and other typical algorithms on MNIST containing noise and underwater multisource data in different scenes shows that the proposed algorithm is superior to others regardless of the efficiency or precision of target classification.

Inference of hidden common driver dynamics by anisotropic self-organizing neural networks.

Batch process monitoring is of great importance to ensure the stable operation during the process running. However, traditional deep learning methods have certain limitations when dealing with complex data structures and dynamic features that are prominent in industrial batch processes. This paper proposes a GraphSAGE-LSTM-based deep canonical correlation analysis (GSA-LSTM-CCA) to enhance the performance of feature learning for batch process monitoring. First, the k-nearest neighbor method is used to construct a graph for the variable-unfolding batch process data as a supplementary structural information for the original spatiotemporal correlations. Then, GraphSAGE and LSTM are used to extract the structural features and time-varying features of data, respectively. Finally, the learned features are weighted and fused to construct a residual generator, and CCA is utilized to carry out process monitoring. The effectiveness and superiority of the proposed method are verified in an actual injection molding process.

Linkage influence of energy market on financial market by multiscale complexity synchronization

This paper investigates attention fatigue detection and multi-grain emotional AI classification for student mental health through an educational brain-computer interface (BCI), with a focus on integrating biomechanical principles to enhance understanding and application. Recognizing the growing importance of students’ mental health and well-being, this study introduces a domain generalization approach in transfer learning to improve cross-subject BCI model accuracy, addressing the challenges of individual variability. The proposed model utilizes only seven electrodes and achieves a 90% accuracy rate in differentiating between two cognitive-behavioral tasks. A truncated weighting algorithm is employed to optimize electrode combinations, enabling effective generalization across subjects. To tackle the practical challenges of emotion recognition in educational settings, the study reduces data sampling points by identifying key brain regions and frequency bands associated with emotions. Machine learning algorithms, including support vector machines (SVM), Bayesian networks, and K-nearest neighbor (KNN), further enhance recognition accuracy. By integrating eye movement and electroencephalography (EEG) signals using deep canonical correlation analysis, the model achieves cumulative accuracy improvements of 15% and 12% compared to unimodal EEG and eye movement data, respectively, across 12 subjects. Incorporating biomechanical principles, the study also examines the mechanical properties of neural tissues and their influence on signal propagation. By analyzing the viscoelastic behavior of brain tissue and its impact on EEG signal transmission, the research provides insights into how mechanical stress and strain affect neural activity. This biomechanical perspective enhances the understanding of individual variability in EEG signals and contributes to the development of more robust and personalized BCI models. The integration of biomechanics with AI-driven emotion classification and attention fatigue detection offers a comprehensive approach to improving student mental health and educational outcomes. This fusion approach demonstrates superior performance in both emotion classification and attention fatigue detection, offering substantial potential for real-time interventions in student mental health and the enhancement of educational outcomes.

Takashima, Naoki Li, Frédéric Grzegorzek, Marcin Shirahama, KimiakiMusic-emotion retrieval is important for treatments of mood disorders and depression based on music. Although existing approaches only investigate one-way retrieval from music to emotions, an approach from emotion to music might be needed for proper application of music-based therapies. This can be achieved by sensor-based music retrieval which firstly recognises a specific emotion based on physiological data acquired by wearable sensors and then identifies music suitable for that emotion. In this paper, we propose Cross-modal Music-emotion Retrieval (CMR) as the first step to achieve retrieval of music based on sensor data. Our approach uses Deep Canonical Correlation Analysis (DeepCCA) which projects music samples and their associated emotion sequences into a common space using deep neural networks, and maximises the correlation between projected music samples and emotion sequences using CCA. Our experiments show the superiority of our approach for CMR compared to one-way retrieval.

Conventional Speech Emotion Recognition (SER) approaches put more emphasis on extracting magnitude spectrum-based features, such as Mel Frequency Cepstral Coefficients (MFCCs), and Mel spectrogram. However, phase information is ignored due to signal processing difficulties such as the phase wrapping issue. This work develops a multichannel Convolution Neural Network-Bidirectional Long Short Term Memory (CNN-BLSTM) architectures with an attention mechanism for speaker-independent SER by considering phase and magnitude spectrum-based features. The phase-based features are extracted using the Modified Group Delay Function (MODGD). The obtained phase features are combined with MFCC features. The CNN-BLSTM network extract learned representation from magnitude and phase features. The learned representation from MFCCs and MODGD are combined and given as an input to the Support Vector Machine (SVM) for classification. The Deep Canonical Correlation Analysis (DCCA) is used to maximize the correlation between magnitude and phase information to improve the conventional SER system’s performance. Here the IEMOCAP database is used for performance analysis. The experimental results show improvement over MFCC features and existing approaches for unimodal SER. In this work, we also developed real-time Web server application for the proposed architecture.

Age-related macular degeneration (AMD) is a prevalent retinal degenerative disease among the elderly and is a major cause of irreversible vision loss worldwide. Although color fundus photography (CFP) and optical coherence tomography (OCT) are widely used for AMD diagnosis, information from a single modal is inadequate to fully capture the complex pathological features of AMD. To address this, this study proposes an innovative multi-modal deep learning framework that fine-tunes pre-trained single-modal retinal models for efficient application in multi-modal AMD categorization tasks. Specifically, two independent vision transformer models are used to extract features from CFP and OCT images, followed by deep canonical correlation analysis (DCCA) to perform nonlinear mapping and fusion of features from both modalities, maximizing cross-modal feature correlation. Moreover, to reduce the computational complexity of multi-modal integration, we introduce the low-rank adaptation (LoRA) technique, which uses low-rank decomposition of parameter matrices, achieving superior performance compared to full fine-tuning with only about 0.49% of the trainable parameters. Experimental results on the public dataset MMC-AMD validate the framework's effectiveness. The proposed model achieves an overall F1-score of 0.948, AUC-ROC of 0.991, and accuracy of 0.949, significantly outperforming existing single-modal and multi-modal baseline models, particularly excelling in recognizing complex pathological categories.

This paper tackles a fine-grained action recognition task that aims to identify pitch-type-specific habits in baseball pitching motions. This task is highly challenging because it requires distinguishing subtle differences among pitch types within the common motion of “throwing.” Although it scarcely addressed in previous studies, it holds great potential for practical applications such as sports performance analysis. We propose a novel multimodal approach that integrates skeleton and RGB modalities. First, the Segment Anything Model (SAM) is employed to remove background information from RGB videos, thereby mitigating background bias. Next, Deep Canonical Correlation Analysis (DCCA) is applied to align the skeleton and RGB features. By further decomposing the RGB features into structural and textural components, the proposed method enhances discriminative performance. Experiments conducted on a custom pitching dataset collected from three pitchers demonstrate that our method outperforms conventional approaches in classification accuracy and effectively captures subtle pitching habits.

With the rapid growth of multimedia content on the Internet, multimedia retrieval has been extensively studied for decades. Inspired by the progress of deep neural network (DNN) in single-media retrieval, the researchers have applied the DNN to cross-media retrieval. Most of the existing methods cast low-level features of cross-media data onto a unified feature space. However, most of these feature spaces usually do not have explicit semantics which results in an unsatisfactory retrieval result. By considering the above issue, cross-media joint representation by hierarchical learning with DSCCA-LR is proposed in this paper which is able to explore jointly the correlation and semantic information. It combines Deep Canonical Correlation Analysis (DCCA) and Logistic Regression (LR) to learn two types of semantic features of of heterogeneous data and concatenate them to form common features for cross modal retrieval. Extensive experiments on two datasets show that the proposed approach achieves superior retrieval accuracy comparing to several state-of-the-art cross-media retrieval methods.

Kernel methods have long played a central role in machine learning by enabling nonlinear modeling in high dimensional feature spaces. Among them, the Gaussian Radial Basis Function (RBF) kernel is widely used due to its smoothness, universality, and excellent performance in supervised learning. In contrast, the Jaccard/Tanimoto coefficient, originally defined over sets and binary vectors, has achieved notable success in chemoinformatics and information retrieval. Recent studies have shown that the Jaccard/Tanimoto coefficient can be extended to real-valued vectors and interpreted as a valid kernel. In this paper, we propose a natural generalization of the Tanimoto kernel by introducing a bandwidth-like parameter, leading to the Bandwidth-Adjustable Tanimoto (BAT) kernel. We focus on the shared structure of the BAT and the Gaussian RBF kernels in terms of their infinite polynomial expansions and investigate the universality of the BAT kernels, clarifying their relationship to the universal Taylor kernels. In particular, we show that by incorporating a bias term, the BAT kernel attains the universal approximation property comparable to that of the Gaussian kernel. Through experiments on classification, regression, and clustering tasks using benchmark datasets, we demonstrate that the BAT kernel performs comparably to the Gaussian kernel, while even the original Tanimoto kernel achieves competitive results. In particular, the BAT kernel attains performance on par with the Gaussian kernel in classification tasks, and it slightly surpasses the Gaussian kernel in regression and clustering tasks. A comparison of their functional forms, together with experiments on both regression and classification tasks involving nonlinear structures, further shows that the BAT kernel yields smoother predictive surfaces than the RBF kernel. Overall, our results indicate that the BAT kernel offers universality and practical performance, making it an alternative to the Gaussian kernel.

Recently, there has been a growing interest in classification of patient samples based on gene expressions. Here the classification task is made more difficult by the noisy nature of the data, and by the overwhelming number of genes relative to the number of available training samples in the data set. Moreover, many of these genes are irrelevant for classification and have negative effect on the accuracy and on the required learning time for the classifier. We propose a new evolutionary computation method to select the most useful subset of genes for molecular classification. We apply this method to three benchmark data sets and present our unbiased experimental results.

Generative Artificial Intelligence (AI) can be seen as a radically new paradigm in the study of data-driven applications, which enables machines to create content on their own in diverse fields, i.e., computer vision, scientific simulations, and natural language processing. A critical issue related to implementing the Generative AI systems is the practical storage, retrieval and control of high-dimensional vectors representations characteristic of the embedding-based AI systems. It is common that older databases cannot scale or provide fast similarity search, which is required in large scale collection of vector operations. The paper examines the concept of integrating Amazon Relational Database Service (RDS) as a vector store to the Generative AI systems, its practicability, performance, and its application to practice. We would propose a new design where the familiar RDS handles such structured relational data that high dimensional vectors can be both stored and read effectively. Our propositions that are founded on advanced indexing techniques, dimensionality reduction, and query optimization operate to indicate that the RDS can be an effective and consistent backend to Generative AI applications. The research paper entails a detailed experiment of text, image, and multimodal embeddings and compares the performance of retrieval, storage efficiency, and computational overhead with special purpose vector databases (such as faiiss and milvus). Conclusions RDS is capable of executing operations of a vector search with very low latency with the appropriate changes and adjustments and can offer a comparatively inexpensive alternative to organizations that already deploy the AWS services. Hybrid systems based on RDS and custom differentiated vector engines are also discussed in the paper to create the most preferable combination of consistency, scalability, and performance. The findings have valuable implications to both researchers and practitioners who may wish to implement Generative AI systems without necessarily incurring the overhead cost of the specialized infrastructure, further encouraging the practical application of AI-based solutions to the industry