Multimodal deep learning for solar radio burst classification

Multimodal learning for facial expression recognition

Solar radio bursts are a kind of instantaneously enhanced radio waves from the Sun. The high-intensity signals can seriously interfere with radio communication and navigation systems. At present, the traditional method is to manually analyze the radio frequency spectrum to find out the solar radio bursts, which has low efficiency and time lag. If the solar radio spectrum can be automatically classified in real time, it will have positive significance for space weather early warning and solar physics research. An automatic classification method of solar radio spectrum using deep learning technology is proposed in this paper. First of all, due to sample imbalance and small number of samples, the data set is expanded according to the characteristics of solar radio spectrogram. Then, spectrum normalization and down-sampling are used to preprocess the solar radio spectrum. Finally, considering the radio spectrum has both the spatial characteristics of the image and the time–frequency characteristics of the sequence signal, convolutional long short-term memory network (ConvLSTM) is used to classify the solar radio spectrum in the paper. ConvLSTM also takes into account the spatial and temporal characteristics of the solar radio spectrogram, which is useful for classifying radio spectrograms that have both of these features. Compared to the method that only considers one of the characteristics, the experimental results show that the method achieves better results than the method using only a single feature.KeywordsSolar radio burstConvLSTMDeep learningImage classification

In solar radio observation, it is inevitable to be disturbed by noise, and the solar radio spectrum is affected. The solar radio spectrum with good quality can provide useful information in analyzing solar activities and improve the accuracy of automatic detection and classification of solar radio bursts. A noise reduction method for solar radio spectrum based on improved guided filter and morphological cascade is proposed in this paper. Firstly, the background noise of solar radio spectrum is subtracted, and the cross-fringe interference caused by channel effect is reduced. Secondly, the improved guided filtering is used to keep the edge of the solar radio burst as far as possible when filtering the burst region of the spectrum. Finally, more noise from other external factors is removed by morphological operation. The experimental results show that the proposed method is effective in removing noise from solar radio spectrum.

4 - Deep multimodal representation learning for noninvasive neural speech decoding

Rapid collection of data sources, varying in volume and structure poses a challenge for scientists to establish a practical approach to manipulating heterogeneous data sources. A multimodal learning and an integrated analysis make it possible to extract much worthwhile information from a collection of multiple simple raw data. Therefore, data integration can lead to a more reliable and robust result. High-throughput sequencing technologies, especially next-generation sequencing, leave us with multi-platform genomic data such as gene expression, SNP, CNV, DNA methylation, and miRNA expression. In this paper, we represented a multimodal deep neural network to exploit the mutual information between three different modalities to classify breast cancer patients into two groups based on their survival rate. Experimental results indicate that our method improves the classification accuracy and performs better on imbalanced data compared to the other single-modal state-of-the-art methods.

In recent years, Deep Learning has been successfully applied to multimodal learning problems, with the aim of learning useful joint representations in data fusion applications. When the available modalities consist of time series data such as video, audio and sensor signals, it becomes imperative to consider their temporal structure during the fusion process. In this paper, we propose the Correlational Recurrent Neural Network (CorrRNN), a novel temporal fusion model for fusing multiple input modalities that are inherently temporal in nature. Key features of our proposed model include: (i) simultaneous learning of the joint representation and temporal dependencies between modalities, (ii) use of multiple loss terms in the objective function, including a maximum correlation loss term to enhance learning of cross-modal information, and (iii) the use of an attention model to dynamically adjust the contribution of different input modalities to the joint representation. We validate our model via experimentation on two different tasks: video-and sensor-based activity classification, and audio-visual speech recognition. We empirically analyze the contributions of different components of the proposed CorrRNN model, and demonstrate its robustness, effectiveness and state-of-the-art performance on multiple datasets.

Deep contrastive representation learning for multi-modal clustering

Multi-modal deep learning networks for RGB-D pavement waste detection and recognition

ABSTRACTThe performance of photovoltaic modules is influenced by solar spectrum even under the same solar irradiance conditions. Spectral factor (SF) is a useful index indicating the ratio of available solar irradiance between actual solar spectrums and the standard AM1·5‐G spectrum. In this study, the influence of solar spectrum on photovoltaic performance in cloudy weather as well as in fine weather is quantitatively evaluated as the reciprocal of SF (SF−1). In the cases of fine weather, the SF−1 suggests that solar spectrum has little influence (within a few %) on the performance of pc‐Si, a‐Si:H/sc‐Si, and copper indium gallium (di)selenide modules, because of the “offset effect”. The performance of a‐Si:H modules and the top layers of a‐Si:H/µc‐Si:H modules can vary by more than ± 10% under the extreme conditions in Japan. The seasonal and locational variations in the SF−1 of the bottom layers are about ± several %. A negative correlation is shown between the top and bottom layers, indicating that the performance of a‐Si:H/µc‐Si:H modules does not exceed the performance, at which the currents of the top and bottom layers are balanced, by the influence of solar spectrum. In the cases of cloudy weather, the SF−1 of the pc‐Si, a‐Si:H/sc‐Si, and copper indium gallium (di)selenide modules is generally higher, suggesting favorable for performance than that in fine weather. Much higher SF−1 than that in fine weather is shown by the a‐Si:H module and the top layer of the a‐Si:H/µc‐Si:H module. The SF−1 of the bottom layer neither simply depend on season nor on location. Copyright © 2011 John Wiley & Sons, Ltd.

Toxicity is paramount for comprehending compound properties, particularly in the early stages of drug design. Due to the diversity and complexity of toxic effects, it became a challenge to compute compound toxicity tasks. To address this issue, we propose a multimodal representation learning model, termed multimodal graph isomorphism network (MMGIN), to address this challenge for compound toxicity multitask learning. Based on fingerprints and molecular graphs of compounds, our MMGIN model incorporates a multimodal representation learning model to acquire a comprehensive compound representation. This model adopts a two-channel structure to independently learn fingerprint representation and molecular graph representation. Subsequently, two feedforward neural networks utilize the learned multimodal compound representation to perform multitask learning, encompassing compound toxicity classification and multiple compound category classification simultaneously. To test the effectiveness of our model, we constructed a novel data set, termed the compound toxicity multitask learning (CTMTL) data set, derived from the TOXRIC data set. We compare our MMGIN model with other representative machine learning and deep learning models on the CTMTL and Tox21 data sets. The experimental results demonstrate significant advancements achieved by our MMGIN model. Furthermore, the ablation study underscores the effectiveness of the introduced fingerprints, molecular graphs, the multimodal representation learning model, and the multitask learning model, showcasing the model's superior predictive capability and robustness.

This paper presents an approach for adaptive scheduling and robustness optimization in global logistics networks by integrating multimodal deep reinforcement learning with Internet of Things (IoT) technologies. We propose an integrated framework comprising a multimodal data fusion mechanism that synthesizes heterogeneous IoT sensor data, historical records, and contextual information; an adaptive deep reinforcement learning architecture that generates dynamic scheduling policies; and a multi-objective robust optimization method that balances operational efficiency with system resilience. The framework addresses key challenges in global logistics including demand volatility, transportation disruptions, and environmental uncertainties. Comprehensive experiments conducted on real-world logistics datasets demonstrate that our approach outperforms traditional methods with an 18.7% reduction in operational costs, 12.4% improvement in service levels, and significantly enhanced robustness under various disruption scenarios. The proposed method maintains 83% performance stability during complex disruptions compared to 51–72% for alternative approaches, while keeping computational requirements feasible for practical deployment. This research demonstrates potential contributions to AI-driven logistics operations management by showing improved supply chain performance through multimodal learning and robust optimization techniques.

Multi-modal clustering (MMC) aims to explore complementary information from diverse modalities for clustering performance facilitating. This article studies challenging problems in MMC methods based on deep neural networks. On one hand, most existing methods lack a unified objective to simultaneously learn the inter- and intra-modality consistency, resulting in a limited representation learning capacity. On the other hand, most existing processes are modeled for a finite sample set and cannot handle out-of-sample data. To handle the above two challenges, we propose a novel Graph Embedding Contrastive Multi-modal Clustering network (GECMC), which treats the representation learning and multi-modal clustering as two sides of one coin rather than two separate problems. In brief, we specifically design a contrastive loss by benefiting from pseudo-labels to explore consistency across modalities. Thus, GECMC shows an effective way to maximize the similarities of intra-cluster representations while minimizing the similarities of inter-cluster representations at both inter- and intra-modality levels. So, the clustering and representation learning interact and jointly evolve in a co-training framework. After that, we build a clustering layer parameterized with cluster centroids, showing that GECMC can learn the clustering labels with given samples and handle out-of-sample data. GECMC yields superior results than 14 competitive methods on four challenging datasets. Codes and datasets are available: https://github.com/xdweixia/GECMC.

Multimodal representations play an important role in multimodal learning tasks, including cross-modal retrieval and intra-modal clustering. However, existing multimodal representation learning approaches focus on building one common space by aligning different modalities and ignore the complementary information across the modalities, such as the intra-modal local structures. In other words, they only focus on the object-level alignment and ignore structure-level alignment. To tackle the problem, we propose a novel symmetric multimodal representation learning framework by transferring local structures across different modalities, namely MTLS. A customized soft metric learning strategy and an iterative parameter learning process are designed to symmetrically transfer local structures and enhance the cluster structures in intra-modal representations. The bidirectional retrieval loss based on multi-layer neural networks is utilized to align two modalities. MTLS is instantiated with image and text data and shows its superior performance on image-text retrieval and image clustering. MTLS outperforms the state-of-the-art multimodal learning methods by up to 32% in terms of R@1 on text-image retrieval and 16.4% in terms of AMI onclustering.

Owing to the limitations of testing methods and the inaccuracy of knowledge, the obtained power transformer information is characterized by fuzziness and randomness, which leads to a lack of accurate description of transformer internal coupling interaction and fault evolution in operation and maintenance departments. Therefore, a thermal state parameter prediction method for a 110 kV transformer based on multi-modal deep learning is designed. The data were resampled and smoothed, and high-frequency noise was removed using a wavelet transform. The running time and spatial temperature rise characteristics of the thermal state parameters of the 110 kV transformer were analyzed, and the variational mode decomposition method was used to decompose the thermal state parameters. By constructing a multimodal deep learning network, abstracted relevant features were extracted from high-dimensional complex input data. By constructing the state equation and measurement equation of the hot spot temperature, a real-time optimal estimation model of the hot spot temperature was established. The extended Kalman filter method was used to build a prediction model for a nonlinear top oil temperature rise system. The experimental results showed that the deviation from the actual value was the least, and the difference between the predicted and measured values of the top oil temperature was relatively small. The prediction time was lower than 0.2s, and the remaining life prediction was correct for 17 units.

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.