Scalable non-negative matrix tri-factorization

In recent years, advancements in artificial intelligence, especially in natural language processing (NLP) models, have progressed rapidly. These models demonstrate remarkable results through training on large datasets and extensive architectures. However, the output process is often a black box, and the decision-making process remains unclear. Our research focuses on the internal representations, specifically the latent variables, generated by NLP models. In earlier work, we explored the latent variables and spaces produced by Sentence-BERT using image generation models. This approach aimed to visualize these spaces by converting discrete textual embeddings into images, introducing continuity and revealing novel relationships. This paper presents the development of an image generation model that uses a common decoder for both GPT-2 and Sentence-BERT, aimed at examining how differences in model architecture affect their latent spaces. We also investigate the impact of training dataset differences by comparing models trained in English and Japanese. Our findings indicate that while the models often generate similar outputs, significant differences emerge in sentences containing multiple elements, attributable to the differing focuses and objectives of the models. Our goal is to understand these latent spaces and contribute to the development of explainable AI.

Multi-view intact space clustering

In recent years, Recommender System (RS) research work has covered a wide variety of Artificial Intelligence techniques, ranging from traditional Matrix Factorization (MF) to complex Deep Neural Networks (DNN). Traditional Collaborative Filtering (CF) recommendation methods such as MF, have limited learning capabilities as it only considers the linear combination between user and item vectors. For learning non-linear relationships, methods like Neural Collaborative Filtering (NCF) incorporate DNN into CF methods. Though, CF methods still suffer from cold start and data sparsity. This paper proposes an improved hybrid-based RS, namely Neural Matrix Factorization++ (NeuMF++), for effectively learning user and item features to improve recommendation accuracy and alleviate cold start and data sparsity. NeuMF++ is proposed by incorporating effective latent representation into NeuMF via Stacked Denoising Autoencoders (SDAE). NeuMF++ can also be seen as the fusion of GMF++ and MLP++. NeuMF is an NCF framework which associates with GMF (Generalized Matrix Factorization) and MLP (Multilayer Perceptrons). NeuMF achieves state-of-the-art results due to the integration of GMF linearity and MLP non-linearity. Concurrently, incorporating latent representations has shown tremendous improvement in GMF and MLP, which result in GMF++ and MLP++. Latent representation obtained through the SDAEs' latent space allows NeuMF++ to effectively learn user and item features, significantly enhancing its learning capability. However, sharing feature extractions among GMF++ and MLP++ in NeuMF++ might hinder its performance. Hence, allowing GMF++ and MLP++ to learn separate features provides more flexibility and greatly improves its performance. Experiments performed on a real-world dataset have demonstrated that NeuMF++ achieves an outstanding result of a test root-mean-square error of 0.8681. In future work, we can extend NeuMF++ by introducing other auxiliary information like text or images. Different neural network building blocks can also be integrated into NeuMF++ to form a more robust recommendation model.

Mass spectrometry imaging techniques measure molecular abundance in a tissue sample at a cellular resolution, all while preserving the spatial structure of the tissue. This kind of technology offers a detailed understanding of the role of several molecular factors in biological systems. For this reason, the development of fast and efficient computational methods that can extract relevant signals from massive experiments has become necessary. A key goal in mass spectrometry data analysis is the identification of molecules with similar functions in the analyzed biological system. This result can be achieved by studying the spatial distribution of the molecules' abundance patterns. To do so, one can perform coclustering, that is, dividing the molecules into groups according to their expression patterns over the tissue and segmenting the tissue according to the molecules' abundance levels. We present TRIFASE, a semi-nonnegative matrix trifactorization technique that performs coclustering while accounting for the spatial correlation of the data. We propose an estimation algorithm that solves the proposed matrix trifactorization problem. Moreover, to improve scalability, we also propose two heuristic approximations of the most expensive steps, which help the algorithm converge while significantly streamlining the computational cost. We validated our method on a series of simulation experiments, comparing the different estimating strategies discussed in the article. Last, we analyzed a mouse brain tissue sample processed with MALDI-MSI technology, showing how TRIFASE extracts specific expression patterns of molecule abundance in localized tissue areas and discovers blocks of proteins whose activation is directly linked to specific biologicalmechanisms.

The necessity to use very large datasets in order to train Generative Adversarial Networks (GANs) has limited their use in cases where the data at disposal are scarce or poorly labelled (e.g., in real life applications). Recently, meta-learning proved that it can help solving effectively few-shot classification problems, but its use in noise-to-image generation was only partially explored. In this paper, we took the first step into applying a meta-learning algorithm (Reptile), to the discriminator of a GAN and to a mapping network in order to optimize the random noise z to guide the generator network into producing images belonging to specific classes. By doing so, we prove that the latent space distribution is crucial for the generation of sharp samples when few training data are at disposal and also managed to generate samples of previously unseen classes just by optimizing the latent space without changing any parameter in the generator network. Finally, we show several experiments with two widely used datasets: MNIST and Omniglot.

Nonnegative matrix tri-factorization and spectral co-clustering are two popular techniques that allow simultaneous clustering of the rows and columns of a matrix. In this paper, by adding a weighting scheme derived from spectral co-clustering into the objective function of nonnegative matrix tri-factorization, we show that the normalized cut information for co-clustering can be incorporated into nonnegative matrix tri-factorization. With the weighting scheme, a weighted nonnegative matrix tri-factorization algorithm for co-clustering is proposed, and extensive experiments show that our method statistically outperforms state-of-the-art co-clustering algorithms.

We suggest a use of genetic programming for transformation from a vector space to an understandable graph representation, which is part of a project to inspect the latent space in matrix factorization. Given a relation matrix, we can apply standard techniques such as non-negative matrix factorization to extract low dimensional latent space in vector representation. While the vector representation of the latent space is useful, it is not intuitive and hard to interpret. The transformation with the help of genetic programming allows us to better understand the underlying latent structure. Applying the method in the context of a stock market, we show that it is possible to recover the tree representation of technical patterns from a relation matrix. Leveraging the properties of the vector representations, we are able to find patterns that correspond to cluster centers of technical patterns. We further investigate the geometry of the latent space.

SCH: Symmetric Consistent Hashing for cross-modal retrieval

Matrix factorization techniques provide efficient tools for the detailed analysis of large-scale biological and biomedical data. While underlying algorithms usually work fully blindly, we propose to incorporate prior knowledge encoded in a graph model. This graph introduces a partial ordering in data without intrinsic (e.g. temporal or spatial) structure, which allows the definition of a graph-autocorrelation function. Using this framework as constraint to the matrix factorization task we develop a second-order source separation algorithm called graph-decorrelation algorithm (GraDe). We demonstrate its applicability and robustness by analyzing microarray data from a stem cell differentiation experiment.

Low-rank matrix factorizations such as Principal Component Analysis (PCA), Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF) are a large class of methods for pursuing the low-rank approximation of a given data matrix. The conventional factorization models are based on the assumption that the data matrices are contaminated stochastically by some type of noise. Thus the point estimations of low-rank components can be obtained by Maximum Likelihood (ML) estimation or Maximum a posteriori (MAP). In the past decade, a variety of probabilistic models of low-rank matrix factorizations have emerged. The most significant difference between low-rank matrix factorizations and their corresponding probabilistic models is that the latter treat the low-rank components as random variables. This paper makes a survey of the probabilistic models of low-rank matrix factorizations. Firstly, we review some probability distributions commonly-used in probabilistic models of low-rank matrix factorizations and introduce the conjugate priors of some probability distributions to simplify the Bayesian inference. Then we provide two main inference methods for probabilistic low-rank matrix factorizations, i.e., Gibbs sampling and variational Bayesian inference. Next, we classify roughly the important probabilistic models of low-rank matrix factorizations into several categories and review them respectively. The categories are performed via different matrix factorizations formulations, which mainly include PCA, matrix factorizations, robust PCA, NMF and tensor factorizations. Finally, we discuss the research issues needed to be studied in the future.

An automated software methodology for biomedical statistics, data pre-processing, and machine learning.

A Structure-Constrained Matrix Factorisation (SCMF) problem is considered where data and structural constraints on one of the matrix factors specific to an application are known. A simple two-step iterative optimisation algorithm can produce unique solutions provided both matrix factors are full-ranked and constraints matrix satisfies certain additional rank conditions. Constraints matrix is apriori known and hence, it can be tested for these rank conditions. Graph theoretical approaches can be used to decompose a graph representing incompatible constraints matrix into compatible subgraphs. However, there is no method available in relevant literature to compute the rank of second matrix factor as it is apriori unknown. Previously, it has been argued that the second matrix factor will naturally be full-ranked, but we show that this is not necessarily true. We develop theoretical bounds on rank of the second matrix factor in terms of ranks of constraints matrix, data matrix and their dimensions. With this new result, uniqueness of a solution can be guaranteed solely based on available constraints and data. Furthermore, we propose Beaded Network Component Analysis algorithm that introduces necessary corrections to the available graph decomposition and mixing algorithms to obtain unique solutions by computing a convex-combination of full-rank factors of subgraphs. The key contributions in this paper are theoretical bounds on rank of a matrix factor and unique solutions to a general SCMF problem.

Matrix Factorization (MF) is widely used in Recommender Systems (RSs) for estimating missing ratings in the rating matrix. MF faces major challenges of handling very sparse and large data. Poisson Factorization (PF) as an MF variant addresses these challenges with high efficiency by only computing on those non-missing elements. However, ignoring the missing elements in computation makes PF weak or incapable for dealing with columns or rows with very few observations (corresponding to sparse items or users). In this work, Metadata-dependent Poisson Factorization (MPF) is invented to address the user/item sparsity by integrating user/item metadata into PF. MPF adds the metadata-based observed entries to the factorized PF matrices. In addition, similar to MF, choosing the suitable number of latent components for PF is very expensive on very large datasets. Accordingly, we further extend MPF to Metadata-dependent Infinite Poisson Factorization (MIPF) that integrates Bayesian Nonparametric (BNP) technique to automatically tune the number of latent components. Our empirical results show that, by integrating metadata, MPF/MIPF significantly outperform the state-of-the-art PF models for sparse and large datasets. MIPF also effectively estimates the number of latent components.

Anomaly detection in smart card logs and distant evaluation with Twitter: a robust framework

Scientific and engineering problems often require the use of artificial intelligence to aid understanding and the search for promising designs. While Gaussian processes (GP) stand out as easy-to-use and interpretable learners, they have difficulties in accommodating big data sets, categorical inputs, and multiple responses, which has become a common challenge for a growing number of data-driven design applications. In this paper, we propose a GP model that utilizes latent variables and functions obtained through variational inference to address the aforementioned challenges simultaneously. The method is built upon the latent-variable Gaussian process (LVGP) model where categorical factors are mapped into a continuous latent space to enable GP modeling of mixed-variable data sets. By extending variational inference to LVGP models, the large training data set is replaced by a small set of inducing points to address the scalability issue. Output response vectors are represented by a linear combination of independent latent functions, forming a flexible kernel structure to handle multiple responses that might have distinct behaviors. Comparative studies demonstrate that the proposed method scales well for large data sets with over 104 data points, while outperforming state-of-the-art machine learning methods without requiring much hyperparameter tuning. In addition, an interpretable latent space is obtained to draw insights into the effect of categorical factors, such as those associated with “building blocks” of architectures and element choices in metamaterial and materials design. Our approach is demonstrated for machine learning of ternary oxide materials and topology optimization of a multiscale compliant mechanism with aperiodic microstructures and multiple materials.