Eccentric regularization: minimizing hyperspherical energy without explicit projection

A deep Koopman operator‐based modelling approach for long‐term prediction of dynamics with pixel‐level measurements

Generative Adversarial Neural Approach Towards Construction of Warli-stick Figure Aboriginal Artwork

Deep clustering incorporates embedding into clustering in order to find a lower-dimensional space suitable for clustering tasks. Conventional deep clustering methods aim to obtain a single global embedding subspace (aka latent space) for all the data clusters. In contrast, in this article, we propose a deep multirepresentation learning (DML) framework for data clustering whereby each difficult-to-cluster data group is associated with its own distinct optimized latent space and all the easy-to-cluster data groups are associated with a general common latent space. Autoencoders (AEs) are employed for generating cluster-specific and general latent spaces. To specialize each AE in its associated data cluster(s), we propose a novel and effective loss function which consists of weighted reconstruction and clustering losses of the data points, where higher weights are assigned to the samples more probable to belong to the corresponding cluster(s). Experimental results on benchmark datasets demonstrate that the proposed DML framework and loss function outperform state-of-the-art clustering approaches. In addition, the results show that the DML method significantly outperforms the SOTA on imbalanced datasets as a result of assigning an individual latent space to the difficult clusters.

Deep Neural Networks have been proved as the go-to approach in modeling data distribution in a latent space, especially in Neural Style Transfer (NST), which casts a specific style extracted from a source image to another target image by calibrating the style and content information in a latent space. While existing methods focuses on different ways to extract features that more precisely describe style or content information to improve existing NST pipelines, the latent space of the NST model has not been well-explored. In this paper, we show that different half-spaces in the latent space are actually associated with particular styles of a network’s generated images. The corresponding constraints of these half-spaces can be computed by using linear classifiers, e.g. a Support Vector Machines (SVM). Leveraging the understanding of the relation between half-spaces in the latent space and output style, we propose the Linear Modification for Latent Representations (LMLR), a method that effectively increases or decreases the level of stylizing in the output image for any given NST model. We empirically evaluate our method on several state-of-the-art NST models and show that LMLR can manipulate the level of stylizing in the output image.

In recent years, advancements in artificial intelligence, especially in natural language processing (NLP) models, have progressed rapidly. These models demonstrate remarkable results through training on large datasets and extensive architectures. However, the output process is often a black box, and the decision-making process remains unclear. Our research focuses on the internal representations, specifically the latent variables, generated by NLP models. In earlier work, we explored the latent variables and spaces produced by Sentence-BERT using image generation models. This approach aimed to visualize these spaces by converting discrete textual embeddings into images, introducing continuity and revealing novel relationships. This paper presents the development of an image generation model that uses a common decoder for both GPT-2 and Sentence-BERT, aimed at examining how differences in model architecture affect their latent spaces. We also investigate the impact of training dataset differences by comparing models trained in English and Japanese. Our findings indicate that while the models often generate similar outputs, significant differences emerge in sentences containing multiple elements, attributable to the differing focuses and objectives of the models. Our goal is to understand these latent spaces and contribute to the development of explainable AI.

Combining autoencoder neural network and Bayesian inversion to estimate heterogeneous permeability distributions in enhanced geothermal reservoir: Model development and verification

We propose to embed time series in a latent space where pairwise Euclidean distances (EDs) between samples are equal to pairwise dissimilarities in the original space, for a given dissimilarity measure. To this end, we use auto-encoder (AE) and encoder-only neural networks to learn elastic dissimilarity measures, e.g., dynamic time warping (DTW), that are central to time series classification (Bagnall et al., 2017). The learned representations are used in the context of one-class classification (Mauceri et al., 2020) on the datasets of UCR/UEA archive (Dau et al., 2019). Using a 1-nearest neighbor (1NN) classifier, we show that learned representations allow classification performance that is close to that of raw data, but in a space of substantially lower dimensionality. This implies substantial and compelling savings in terms of computational and storage requirements for nearest neighbor time series classification.

In real-world sequential decision making tasks like autonomous driving, robotics, and healthcare, learning from observed state-action trajectories is critical for tasks like imitation, classification, and clustering. For example, self-driving cars must replicate human driving behaviors, while robots and healthcare systems benefit from modeling decision sequences, whether or not they come from expert data. Existing trajectory encoding methods often focus on specific tasks or rely on reward signals, limiting their ability to generalize across domains and tasks. Inspired by the success of embedding models like CLIP and BERT in static domains, we propose a novel method for embedding state-action trajectories into a latent space that captures the skills and competencies in the dynamic underlying decision-making processes. This method operates without the need for reward labels, enabling better generalization across diverse domains and tasks. Our contributions are threefold: (1) We introduce a trajectory embedding approach that captures multiple abilities from state-action data. (2) The learned embeddings exhibit strong representational power across downstream tasks, including imitation, classification, clustering, and regression. (3) The embeddings demonstrate unique properties, such as controlling agent behaviors in IQ-Learn and an additive structure in the latent space. Experimental results confirm that our method outperforms traditional approaches, offering more flexible and powerful trajectory representations for various applications. Our code is available at https://github.com/Erasmo1015/vte.

A novel damage detection approach based on Auto-encoder neural network is proposed to identify damage in beam-like bridges subjected to a moving mass. In this approach, several sensors are used to measure structural vibration responses during a mass moving across the bridge. An auto-encoder (AE) neural network is designed to extract features from the measured responses. A fixed moving window is used to cut out the time-domain responses to generate inputs of the AE neural network. Moreover, some constraints are applied on the hidden layer to improve the performance of the AE network in training process. When the training is complete, the encoder was regarded as a feature extractor. And the damage index is defined as the cosine distance between two feature vectors obtained from adjacent data windows. By moving the window along the measured vibration data, we can calculate a damage index series and locate the damage position of the structure. To demonstrate the performance of the proposed method, numerical simulation is carried out. The results show that the proposed method can accurately locate both single and multiple damages using acceleration response. It infers the proposed method is promising for structural damage detection.

Glasses offer a broad range of tunable thermophysical properties that are linked to their compositions. However, it is challenging to establish a universal composition-property relationship of glasses due to their enormous compositions and chemical space. Here, we address this problem and develop a metamodel of the composition-atomistic structure relationship of a class of glassy materials via a machine learning (ML) approach. Within this ML framework, an unsupervised deep learning technique, viz., a convolutional neural network (CNN) autoencoder, and a regression algorithm, viz. random forest (RF), are integrated into a fully automated pipeline to predict the spatial distribution of atoms in a glass. The RF regression model predicts the pair correlation function of a glass in a latent space. Subsequently, the decoder of the CNN converts the latent space representation to the actual pair correlation function of the given glass. The atomistic structures of silicate (SiO2) and sodium borosilicate (NBS) based glasses with varying compositions and dopants are collected from molecular dynamics (MD) simulations to establish and validate this ML pipeline. The model is found to predict the atom pair correlation functions for many unknown glasses very accurately. This method is very generic and can accelerate the design, discovery, and fundamental understanding of the composition-atomistic structure relationship of glasses and other materials.

Quantification, or prevalence estimation, is the task of predicting the prevalence of each class within an unknown bag of examples. Most existing quantification methods in the literature rely on prior probability shift assumptions to create a quantification model that uses the predictions of an underlying classifier to make optimal prevalence estimates. In this work, we present an end-to-end neural network that uses Gaussian distributions in latent spaces to obtain invariant representations of bags of examples. This approach addresses the quantification problem using deep learning, enabling the optimization of specific loss functions relevant to the problem and avoiding the need for an intermediate classifier, tackling the quantification problem as a direct optimization problem. Our method achieves state-of-the-art results, both against traditional quantification methods and other deep learning approaches for quantification. The code needed to reproduce all our experiments is publicly available at https://github.com/AICGijon/gmnet.

Deep neural networks are widely used in computer vision for image classification, segmentation and generation. They are also often criticised as “black boxes” because their decision-making process is often not interpretable by humans. However, learning explainable representations that explicitly disentangle the underlying mechanisms that structure observational data is still a challenge. To further explore the latent space and achieve generic processing, we propose a pipeline for discovering the explainable directions in the latent space of generative models. Since the latent space contains semantically meaningful directions and can be explained, we propose a pipeline to fully resolve the representation of the latent space. It consists of a Dirichlet encoder, conditional deterministic diffusion, a group-swap and a latent traversal module. We believe that this study provides an insight into the advancement of research explaining the disentanglement of neural networks in the community.

Physical systems governed by advection-dominated partial differential equations (PDEs) are found in applications ranging from engineering design to weather forecasting. They are known to pose severe challenges to both projection-based and non-intrusive reduced order modeling, especially when linear subspace approximations are used. In this work, we develop an advection-aware (AA) autoencoder network that can address some of these limitations by learning efficient, physics-informed, nonlinear embeddings of the high-fidelity system snapshots. A fully non-intrusive reduced order model is developed by mapping the high-fidelity snapshots to a latent space defined by an AA autoencoder, followed by learning the latent space dynamics using a long-short-term memory (LSTM) network. This framework is also extended to parametric problems by explicitly incorporating parameter information into both the high-fidelity snapshots and the encoded latent space. Numerical results obtained with parametric linear and nonlinear advection problems indicate that the proposed framework can reproduce the dominant flow features even for unseen parameter values.

Accurate modeling of disease progression is essential for comprehending the heterogeneous neuropathologies such as Alzheimer's Disease (AD). Traditional neuroimaging analysis often confound disease effects with normal aging, complicating the differential diagnosis. Recent advancements in deep learning have catalyzed the development of disentanglement techniques in Autoencoder networks, aiming to segregate longitudinal changes attributable to aging from those due to disease-specific alterations within the latent space. However, existing longitudinal disentanglement methods usually model disease as a single axis factor which ignores the complexity and heterogeneity of Alzheimer's Disease. In response to this issue, we propose a novel Surface-based Multi-axis Disentanglement framework.This framework posits multiple disease axes within the latent space, enhancing the model's capacity to encapsulate the multifaceted nature of AD, which includes various disease trajectories. To assign axes to data trajectories without explicit ground truth labels, we implement a longitudinal contrastive loss leveraging self-supervision, thereby refining the separation of disease trajectories. Evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset ( ), our model demonstrates superior performance in delineating between cognitively normal (CN), mild cognitive impairment (MCI), and AD subjects,classification of stable MCI vs converting MCI and Amyloid status, compared to the single-axis model. This is further substantiated through an ablation study on the contrastive loss, underscoring the utility of our multi-axis approach in capturing the complex progression patterns of AD. The code is available at: https://github.com/jianweizhang17/MultiAxisDisentanglement.git.

Robust Latent Regression with discriminative regularization by leveraging auxiliary knowledge