StructEdit: Learning Structural Shape Variations

Accurate modeling of disease progression is essential for comprehending the heterogeneous neuropathologies such as Alzheimer's Disease (AD). Traditional neuroimaging analysis often confound disease effects with normal aging, complicating the differential diagnosis. Recent advancements in deep learning have catalyzed the development of disentanglement techniques in Autoencoder networks, aiming to segregate longitudinal changes attributable to aging from those due to disease-specific alterations within the latent space. However, existing longitudinal disentanglement methods usually model disease as a single axis factor which ignores the complexity and heterogeneity of Alzheimer's Disease. In response to this issue, we propose a novel Surface-based Multi-axis Disentanglement framework.This framework posits multiple disease axes within the latent space, enhancing the model's capacity to encapsulate the multifaceted nature of AD, which includes various disease trajectories. To assign axes to data trajectories without explicit ground truth labels, we implement a longitudinal contrastive loss leveraging self-supervision, thereby refining the separation of disease trajectories. Evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset ( ), our model demonstrates superior performance in delineating between cognitively normal (CN), mild cognitive impairment (MCI), and AD subjects,classification of stable MCI vs converting MCI and Amyloid status, compared to the single-axis model. This is further substantiated through an ablation study on the contrastive loss, underscoring the utility of our multi-axis approach in capturing the complex progression patterns of AD. The code is available at: https://github.com/jianweizhang17/MultiAxisDisentanglement.git.

Collisional-radiative (CR) models describe the atomic processes in a plasma by tracking the population density in the ground and excited states for each charge state of the atom/ion. These models predict important plasma properties such as charge state distributions and radiative emissivity and opacity. Accurate descriptions of the CR balance of the plasma are essential in fusion whole device modeling, especially when significant impurities are introduced into the plasmas. In an integrated fusion plasma and CR simulation, a CR model, which is a high-dimensional stiff ODE, needs to be solved on each grid point in the configuration space, and can overwhelm the overall computational cost. In this work, we propose a machine-learning-based method that discovers a latent space and learns its corresponding latent dynamics, which can capture the essential physics to make accurate predictions at much lower online computational cost. The proposed approach is physics-assisted, due to its combination of a physical latent space and a data-driven latent space. It has been demonstrated that the proposed architecture can predict both the full-order CR dynamics and a physical quantity of interest accurately.

Distribution-motivated 3D Style Characterization Based on Latent Feature Decomposition

Named Entity Recognition (NER) is a fundamental task in information extraction from unstructured text. Most previous machine-learning-based NER systems are domain-specific, which implies that they may only perform well on some specific domains (e.g., Newswire ) but tend to adapt poorly to other related but different domains (e.g., Weblog ). Recently, transfer learning techniques have been proposed to NER. However, most transfer learning approaches to NER are developed for binary classification, while NER is a multiclass classification problem in nature. Therefore, one has to first reduce the NER task to multiple binary classification tasks and solve them independently. In this article, we propose a new transfer learning method, named Transfer Joint Embedding (TJE), for cross-domain multiclass classification, which can fully exploit the relationships between classes (labels), and reduce domain difference in data distributions for transfer learning. More specifically, we aim to embed both labels (outputs) and high-dimensional features (inputs) from different domains (e.g., a source domain and a target domain) into a unified low-dimensional latent space, where 1) each label is represented by a prototype and the intrinsic relationships between labels can be measured by Euclidean distance; 2) the distance in data distributions between the source and target domains can be reduced; 3) the source domain labeled data are closer to their corresponding label-prototypes than others. After the latent space is learned, classification on the target domain data can be done with the simple nearest neighbor rule in the latent space. Furthermore, in order to scale up TJE, we propose an efficient algorithm based on stochastic gradient descent (SGD). Finally, we apply the proposed TJE method for NER across different domains on the ACE 2005 dataset, which is a benchmark in Natural Language Processing (NLP). Experimental results demonstrate the effectiveness of TJE and show that TJE can outperform state-of-the-art transfer learning approaches to NER.

Latent Embedded Graphs for Image and Shape Interpolation

Earth observation (EO) yields large-scale, multimodal datasets collected from various satellite missions, DEMs, land-use data, and textual metadata. Foundation models like 4M (Massively Multimodal Masked Modeling) can learn a joint embedding space that bridges modality gaps, mitigates missing data issues, and facilitates partial spatio-temporal alignment [1]. However, directly training such foundation models on the vast, high-dimensional original EO datasets is not only computationally intensive but also imposes substantial demands on storage resources.To address this, one can leverage VQ-VAE (Vector Quantized-Variational AutoEncoder) as neural compressors to transform high-dimension multimodal inputs into a few discrete indices, significantly reducing data volume while preserving critical information. By inverting the tokenization process, we can reconstruct the original high-dimensional data with minimal quality loss, aided by adversarial and perceptual losses that enhance reconstruction fidelity.Traditional VQ-based approaches, however, face challenges such as inefficient codebook utilization and limited latent space representation. To overcome these, we propose scaling strategies that complement 4M’s tokenizer-based architecture. By expanding the codebook size, latent dimensions, and network depth, our method captures the complexity of EO modalities more effectively. Specifically, we employ spherical quantization techniques like Grouped Spherical Quantization (GSQ) to address limitations in traditional approaches [2]. GSQ constrains codebook vectors to a spherical surface, stabilizing training, preventing code collapse, and promoting uniform codebook usage. Unlike standard VQ, GSQ uses spherical initialization and normalization to maintain consistent distances among codebook entries, ensuring robust latent space coverage even under extreme compression or large codebooks. From our empirical and ablation studies, alternative methods like LFQ (Lookup-Free Quantization), FSQ (Finite Scalar Quantization), and RVQ (Residual Vector Quantizer) often exhibit limitations, such as tightly coupling the latent dimension to codebook size or relying on specialized training losses. In contrast, spherical-based techniques effectively decouple latent dimensions from codebook vocabulary, providing greater flexibility and scalability as data demands increase.Our approach enables neural compressors to adapt to varying scales of compression and complexity without compromising performance. Comprehensive scalability experiments—examining large codebooks, deeper networks, and diverse compression ratios—assessed the generalizability of the proposed compression strategies and demonstrated their effectiveness on high-dimensional, large-scale EO data with minimal information loss. By integrating advanced compression techniques with scalable architectures, this framework establishes a robust foundation for addressing multimodal challenges in EO research that significantly reduces the difficulty of training foundation models on multimodal high-dimensional EO data.

Object detection has made remarkable progress in recent years. While in real-world scenarios, a model is expected to generalize to novel objects that it never explicitly trained on. Though pre-trained vision language model has shown powerful results in zero-shot classification task, adapting it to detection task is non-trivial due to the detection includes region-level reasoning as well as non-semantic localization. In this dissertation, a method built on detr-style architecture and contrastive dis- tillation has been proposed. It utilizes the CLIP model to provide semantic-rich features as priors for querying novel objects. Besides, the model is trained to align with CLIP in a latent space via contrastive loss, enabling it to distinguish unseen classes. The effectiveness of the proposed method is supported by the experimental results with 65.3 novel AR and 23.4 novel mAP on MSCOCO dataset. Its variants out- performs its counter part by 3.5 mAP and 3.1 mAP respectively. The proposed contrastive distillation loss could also be integrated with other framework and achieves the best performance. The significance of different modules is revealed through ablation study and visualization study. The qualitative analysis demonstrates the potential of the proposed method as an effective on-the-fly detector. In final part, a discussion section analyzes the critical factors that contribute to open vocabulary object detection. It provides a unified perspective on reconstruction loss and contrastive loss, offering an interpretation of feature transfer within the context of open vocabulary scenarios.

Graph, as an important data representation, is ubiquitous in many real world applications ranging from social network analysis to biology. How to correctly and effectively learn and extract information from graph is essential for a large number of machine learning tasks. Graph embedding is a way to transform and encode the data structure in high dimensional and non-euclidean feature space to a low dimensional and structural space, which is easily exploited by other machine learning algorithms. We have witnessed a huge surge of such embedding methods, from statistical approaches to recent deep learning methods such as the graph convolutional networks (GCN). Deep learning approaches usually outperform the traditional methods in most graph learning benchmarks by building an end-to-end learning framework to optimize the loss function directly. However, most of the existing GCN methods can only perform convolution operations with node features, while ignoring the handy information in edge features, such as relations in knowledge graphs. To address this problem, we present CensNet, Convolution with Edge-Node Switching graph neural network, for learning tasks in graph-structured data with both node and edge features. CensNet is a general graph embedding framework, which embeds both nodes and edges to a latent feature space. By using line graph of the original undirected graph, the role of nodes and edges are switched, and two novel graph convolution operations are proposed for feature propagation. Experimental results on real-world academic citation networks and quantum chemistry graphs show that our approach achieves or matches the state-of-the-art performance in four graph learning tasks, including semi-supervised node classification, multi-task graph classification, graph regression, and link prediction.

Much efforts have lately been concentrated on increasing the precision of recommendations following the Netflix Prize competition. Recently, many researchers and industries have noted that other factors like adequate presentation of the results can add more utility to a recommender system than slight improvement in the precision. In this paper, we suggest a methodology for user-friendly representation of recommendations to the end users. Our scheme unifies the two objectives of prediction and visualization in the core of a unique approach. Users and items are first embedded into a high dimensional latent feature space according to a predictor function, particularly designated to meet visualization requirements. The data is then projected into a 2-dimensional space by Curvilinear Component Analysis (CCA). CCA draws personalized Item Maps (PIMs) representing a small subset of items to the active user. The intra-item semantic correlations are preserved in PIMs which is inherited from the clustering property of the high-dimensional embedding space. Our prediction function and the projection method are both non-linear to increase the clarity of the maps and to limit the effect of projection error. The algorithms are tested on three versions of the MovieLens dataset and the Netflix dataset to show they combine good accuracy with satisfactory visual properties. Moreover, a novel offline methodology is suggested to measure the quality of PIMs.

We propose a novel construction for extracting a central or limit shape in a shape collection, connected via a functional map network. Our approach is based on enriching the latent space induced by a functional map network with an additional natural metric structure. We call this shape‐like dual object the limit shape and show that its construction avoids many of the biases introduced by selecting a fixed base shape or template. We also show that shape differences between real shapes and the limit shape can be computed and characterize the unique properties of each shape in a collection – leading to a compact and rich shape representation. We demonstrate the utility of this representation in a range of shape analysis tasks, including improving functional maps in difficult situations through the mediation of limit shapes, understanding and visualizing the variability within and across different shape classes, and several others. In this way, our analysis sheds light on the missing geometric structure in previously used latent functional spaces, demonstrates how these can be addressed and finally enables a compact and meaningful shape representation useful in a variety of practical applications.

This work tackles the important task of understanding out-of-distribution behavior in two prominent types of generative models, i.e., GANs and Diffusion models. Understanding this behavior is crucial in understanding their broader utility and risks as these systems are increasingly deployed in our daily lives. Our first contribution is demonstrating that diffusion spaces outperform GANs' latent spaces in inverting high-quality OOD images. We also provide a theoretical analysis attributing this to the lack of prior holes in diffusion spaces. Our second significant contribution is to provide a theoretical hypothesis that diffusion spaces can be projected onto a bounded hypersphere, enabling image manipulation through geodesic traversal between inverted images. Our analysis shows that different geodesics share common attributes for the same manipulation, which we leverage to perform various image manipulations. We conduct thorough empirical evaluations to support and validate our claims. Finally, our third and final contribution introduces a novel approach to the few-shot sampling for out-of-distribution data by inverting a few images to sample from the cluster formed by the inverted latents. The proposed technique achieves state-of-the-art results for the few-shot generation task in terms of image quality. Our research underscores the promise of diffusion spaces in out-of-distribution imaging and offers avenues for further exploration. Please find more details about our project at \url{http://cvit.iiit.ac.in/research/projects/cvit-projects/diffusionOOD}

Deep learning has given AI-based methods for music creation a boost by over the past years. An important challenge in this field is to balance user control and autonomy in music generation systems. In this work, we present BassNet, a deep learning model for generating bass guitar tracks based on musical source material. An innovative aspect of our work is that the model is trained to learn a temporally stable two-dimensional latent space variable that offers interactive user control. We empirically show that the model can disentangle bass patterns that require sensitivity to harmony, instrument timbre, and rhythm. An ablation study reveals that this capability is because of the temporal stability constraint on latent space trajectories during training. We also demonstrate that models that are trained on pop/rock music learn a latent space that offers control over the diatonic characteristics of the output, among other things. Lastly, we present and discuss generated bass tracks for three different music fragments. The work that is presented here is a step toward the integration of AI-based technology in the workflow of musical content creators.

MGAug: Multimodal Geometric Augmentation in Latent Spaces of Image Deformations.

Networks and point processes provide flexible tools for representing and modeling complex dependencies in data arising from various social and physical domains. Graphs, or networks, encode relational dependencies between entities, while point processes characterize temporal or spatial interactions among events.<br>In the first part of this dissertation, we consider dynamic network data (such as communication networks) in which links connecting pairs of nodes appear continuously over time. We propose latent space point process models to capture two different aspects of the data: (i) communication occurs at a higher rate between individuals with similar latent attributes (i.e., homophily); and (ii) individuals tend to reciprocate communications from others, but in a varied manner. Our framework marries ideas from point process models, including Poisson and Hawkes processes, with ideas from latent space models of static networks. We evaluate our models on several real-world datasets and show that a dual latent space model, which accounts for heterogeneity in both homophily and reciprocity, significantly improves performance in various link prediction and network embedding tasks.<br>In the second part of this dissertation, we develop nonparametric goodness-of-fit tests for discrete distributions and point processes that contain intractable normalization constants, providing the first generally applicable and computationally feasible approaches under those circumstances. Specifically, we propose and characterize Stein operators for discrete distributions, and construct a general Stein operator for point processes using the Papangelou conditional intensity function. Based on the proposed Stein operators, we establish kernelized Stein discrepancy measures for discrete distributions and point processes, which enable us to develop nonparametric goodness-of-fit tests for un-normalized density/intensity functions. We apply the kernelized Stein discrepancy tests to discrete distributions (including network models) as well as temporal and spatial point processes. Our experiments demonstrate that the proposed tests typically outperform two-sample tests based on the maximum mean discrepancy, which, unlike our goodness-of-fit tests, assume the availability of exact samples from the null model.<br>

In this paper, a kernel-based nonlinear version of the oblique subspace projection (OBSP) operator is defined in terms of kernel functions. Input data are implicitly mapped into a high-dimensional kernel feature space by a nonlinear mapping, which is associated with a kernel function. The OBSP expression is then derived in the feature space, which is kernelized in terms of the kernel functions in order to avoid explicit computation in the high-dimensional feature space. The resulting kernelized OBSP algorithm is equivalent to a nonlinear OBSP in the original input space. Experimental results based on simulated hyperspectral data and real hyperspectral imagery shows that the kernel oblique subspace projection (KOBSP) outperforms the conventional OBSP.