Semantically consistent multi-view representation learning

Multiview data, characterized by rich features, are crucial in many machine learning applications. However, effectively extracting intraview features and integrating interview information present significant challenges in multiview learning (MVL). Traditional deep network-based approaches often involve learning multiple layers to derive latent. In these methods, the features of different classes are typically implicitly embedded rather than systematically organized. This lack of structure makes it challenging to explicitly map classes to independent principal subspaces in the feature space, potentially causing class overlap and confusion. Consequently, the capability of these representations to accurately capture the intrinsic structure of the data remains uncertain. In this article, we introduce an innovative multiview representation learning (MVRL) by maximizing two information-theoretic metrics: intraview coding rate reduction and interview mutual information. Specifically, in the intraview representation learning, we aim to optimize feature representations by maximizing the coding rate difference between the entire dataset and individual classes. This process expands the feature representation space while compressing the representations within each class, resulting in more compact feature representations within each viewpoint. Subsequently, we align and fuse these view-specific features through space transformation and cross-sample fusion to achieve consistent representation across multiple views. Finally, we maximize information transmission to maintain consistency and correlation among data representations across views. By maximizing mutual information between the consensus representations and view-specific representations, our method ensures that the learned representations capture more concise intrinsic features and correlations among different views, thereby enhancing the performance and generalization ability of MVL. Experiments show that the proposed methods have achieved excellent performance.

In real-world applications, clustering or classification can usually be improved by fusing information from different views. Therefore, unsupervised representation learning on multi-view data becomes a compelling topic in machine learning. In this paper, we propose a novel and flexible unsupervised multi-view representation learning model termed Collaborative Multi-View Information Bottleneck Networks (CMIB-Nets), which comprehensively explores the common latent structure and the view-specific intrinsic information, and discards the superfluous information in the data significantly improving the generalization capability of the model. Specifically, our proposed model relies on the information bottleneck principle to integrate the shared representation among different views and the view-specific representation of each view, prompting the multi-view complete representation and flexibly balancing the complementarity and consistency among multiple views. We conduct extensive experiments (including clustering analysis, robustness experiment, and ablation study) on real-world datasets, which empirically show promising generalization ability and robustness compared to state-of-the-arts.

<div class="section abstract"><div class="htmlview paragraph">Cooperative perception has attracted wide attention given its capability to leverage shared information across connected automated vehicles (CAVs) and smart infrastructure to address the occlusion and sensing range limitation issues. To date, existing research is mainly focused on prototyping cooperative perception using only one type of sensor such as LiDAR and camera. In such cases, the performance of cooperative perception is constrained by individual sensor limitations. To exploit the multi-modality of sensors to further improve distant object detection accuracy, in this paper, we propose a unified multi-modal multi-agent cooperative perception framework that integrates camera and LiDAR data to enhance perception performance in intelligent transportation systems. By leveraging the complementary strengths of LiDAR and camera sensors, our framework utilizes the geometry information from LiDAR and the semantic information from cameras to achieve an accurate cooperative perception system. In order to fuse the multi-agent and multi-modal features, we use a bird’s-eye view (BEV) space as the consistent and unified feature representations and employ a transformer-based network for effective multi-agent multi-modal BEV feature fusion. We validate our method on the OPV2V and V2XSim benchmarks, achieving state-of-the-art performance in 3D cooperative perception tasks. The proposed framework significantly improves object detection accuracy and robustness, especially in complex traffic scenarios with occlusions such as dense intersections.</div></div>

Deep multi-view representation learning focuses on training a unified low-dimensional representation for data with multiple sources or modalities. With the rapidly growing attention of graph neural networks, more and more researchers have introduced various graph models into multi-view learning. Although considerable achievements have been made, most existing methods usually propagate information in a single view and fuse multiple information only from the perspective of attributes or relationships. To solve the aforementioned problems, we propose an efficient model termed Dual Fusion-Propagation Graph Neural Network (DFP-GNN) and apply it to deep multi-view clustering tasks. The proposed method is designed with three submodules and has the following merits: a) The proposed view-specific and cross-view propagation modules can capture the consistency and complementarity information among multiple views; b) The designed fusion module performs multi-view information fusion with the attributes of nodes and the relationships among them simultaneously. Experiments on popular databases show that DFP-GNN achieves significant results compared with several state-of-the-art algorithms.

Prestack seismic data can reveal subtle differences in reservoir lithology and fluid characteristics, making it invaluable for hydrocarbon exploration. With the advancement of deep learning technologies, intelligent prestack interpretation methods have gained widespread application. However, existing methods generally suffer from two major limitations: first, in feature extraction, current models struggle to adequately capture the complex inter-trace variations and offset-dependent correlations in prestack data, resulting in insufficient information utilization; second, in scenario adaptation, due to the lack of unified foundational feature representations, each new task requires designing and training models from scratch, leading to high computational costs and low generalization efficiency. Foundation model technology, leveraging large-scale pretraining and efficient fine-tuning mechanisms, can establish unified deep feature representations to enhance inter-trace variation modeling capabilities and achieve rapid application adaptation through the pretrain-finetune paradigm. However, existing seismic foundation models primarily target poststack data and face two major challenges when applied to prestack data interpretation: lack of masking mechanisms specifically designed for inter-trace variation characteristics in prestack data; absence of physical constraints in model architectures that emphasize the importance of inter-trace variations. To address these challenges, this paper proposes a foundation model framework specifically designed for prestack seismic data. First, by randomly masking portions of offset data, the framework effectively models inter-trace variation characteristics and their physical correlations. Second, sparse constraint mechanisms are introduced to highlight key physical features such as AVO responses. Finally, lightweight adapter modules are employed to achieve rapid scenario adaptation. Synthetic data experiments validate the effectiveness of the masking reconstruction mechanism and physical constraint modules. Field data validation, using lithology identification and gas-bearing detection as typical applications, demonstrates that the framework achieves favorable results in both applications, providing a unified and efficient solution for intelligent interpretation of prestack seismic data.

In conventional unsupervised multi-view clustering (MVC), learning of representations from heterogeneous multiview data and its subsequent clustering are often separately optimized. The disparate optimization would lead to suboptimal performance because multi-view representation learning is not goal-directed. In this paper, we unify unsupervised multi-view learning and deep clustering in a novel discriminative Self-supervised Deep Correlational Multi-view Clustering (SDC-MVC) network. A new unified loss function is proposed to incorporate consensus information into discriminative representations, in which, the former is learnt by maximizing the canonical correlation among multi-view representations projected by neural networks, and the later is achieved through using confident clustering assignments as supervision. Further, multi-view representations are harnessed by our proposed Deep Serial Feature-level (DSF) Fusion layer. Experiments on three public datasets demonstrated that our method outperforms six state-of-the-art correlation-based MVC algorithms in terms of three evaluation metrics.

Adversarial correlated autoencoder for unsupervised multi-view representation learning

In this paper, we propose a deep learning model based on deep multi-view representation learning, to address the video object segmentation task. The proposed model emphasizes the importance of the inherent correlation between video frames and incorporates a multi-view representation learning based on deep canonically correlated autoencoders. The multi-view representation learning in our model provides an efficient mechanism for capturing inherent correlations by jointly extracting useful features and learning better representation into a joint feature space, i.e., shared representation. To increase the training data and the learning capacity, we train the proposed model with pairs of video frames, i.e., Fa and Fb. During the segmentation phase, the deep canonically correlated autoencoders model encodes useful features by processing multiple reference frames together, which is used to detect the frequently reappearing. Our model enhances the state-of-the-art deep learning-based methods that mainly focus on learning discriminative foreground representations over appearance and motion. Experimental results over two large benchmarks demonstrate the ability of the proposed method to outperform competitive approaches and to reach good performances, in terms of semantic segmentation.

Facial expression recognition (FER) in uncontrolled, in-the-wild settings is a challenging task due to variations in pose, illumination, occlusion, and background clutter. The state-of-the-art methods on FER employ fixed size pooling in attention networks, resulting in suboptimal robustness in unconstrained environments. In this work, a novel Multi-Scale Attention Network (MSAN) is integrated with the YOLOv8 backbone architecture for expression classification. The YOLOv8 backbone provides efficient feature extraction with a lightweight, detection-oriented architecture, enabling real-time processing and strong spatial feature representation. The Convolutional Block Attention Mechanism (CBAM) is employed in the proposed work. The standard convolutional blocks in YOLOv8 backbone are replaced with MSAN, wherein the fixed-size pooling in the spatial attention mechanism of CBAM is replaced with a Pyramid Pooling Mechanism (PPM). The PPM employs adaptive average pooling at multiple resolutions to capture multi-scale contextual information, which is subsequently fused into a unified feature representation. This enables the spatial attention mechanism in MSAN to capture both global facial context and local region-level cues simultaneously, leading to superior robustness to scale variations, partial occlusions, and diverse expression variations. The work is evaluated on benchmark wild datasets including RAF-DB, FER2013, and AffectNet, with all experiments performed using 5-fold cross-validation. Experimental results demonstrate that the multi-scale attention network with PPM in YOLOv8 backbone achieved improvement in classification accuracy and robustness when compared with the state-of-the-art methods for expression classification in the wild. These findings highlight the effectiveness of integrating MSAN into lightweight detection-based backbones for real-world FER applications.

Multi-view Low-rank Preserving Embedding: A novel method for multi-view representation

Noises and outliers often degrade the final prediction performance in practical data processing. Multi-view learning by integrating complementary information across heterogeneous modalities has become one of the core techniques in the field of machine learning. However, existing methods rely on explicit-view clustering and stringent alignment assumptions, which affect the effectiveness in addressing the challenges such as inconsistencies between views, noise interference, and misalignment across different views. To alleviate these issues, we present a latent multi-view representation learning model based on low-rank embedding by implicitly uncovering the latent consistency structure of data, which allows us to achieve robust and efficient multi-view feature fusion. In particular, we utilize low-rank constraints to construct a unified latent subspace representation and introduce an adaptive noise suppression mechanism that significantly enhances robustness against outliers and noise interference. Moreover, the Augmented Lagrangian Multiplier Alternating Direction Minimization (ALM-ADM) framework enables efficient optimization of the proposed method. Experimental results on multiple benchmark datasets demonstrate that the proposed approach outperforms existing state-of-the-art methods in both clustering performance and robustness.

A multi-view representation learning framework for commonsense knowledge bases

The existing Multi-View Learning (MVL) is to discuss how to learn from patterns with multiple information sources and has been proven its superior generalization to the usual Single-View Learning (SVL). However, in most real-world cases there are just single source patterns available such that the existing MVL cannot work. The purpose of this paper is to develop a new multi-view regularization learning for single source patterns. Concretely, for the given single source patterns, we first map them into M feature spaces by M different empirical kernels, then associate each generated feature space with our previous proposed Discriminative Regularization (DR), and finally synthesize M DRs into one single learning process so as to get a new Multi-view Discriminative Regularization (MVDR), where each DR can be taken as one view of the proposed MVDR. The proposed method achieves: (1) the complementarity for multiple views generated from single source patterns; (2) an analytic solution for classification; (3) a direct optimization formulation for multi-class problems without one-against-all or one-against-one strategies.

A deep multi-view imbalanced learning approach for identifying informative COVID-19 tweets from social media

Multi-view representation learning with Kolmogorov-Smirnov to predict default based on imbalanced and complex dataset