On Reductions and Representations of Learning Problems in Euclidean Spaces

We consider the problem of image representation and clustering. Traditionally, an n1 x n2 image is represented by a vector in the Euclidean space R n1 x n2. Some learning algorithms are then applied to these vectors in such a high dimensional space for dimensionality reduction, classification, and clustering. However, an image is intrinsically a matrix, or the second order tensor. The vector representation of the images ignores the spatial relationships between the pixels in an image. In this paper, we introduce a tensor framework for image analysis. We represent the images as points in the tensor space Rn1 mathcal Rn2 which is a tensor product of two vector spaces. Based on the tensor representation, we propose a novel image representation and clustering algorithm which explicitly considers the manifold structure of the tensor space. By preserving the local structure of the data manifold, we can obtain a tensor subspace which is optimal for data representation in the sense of local isometry. We call it TensorImage approach. Traditional clustering algorithm such as k-means is then applied in the tensor subspace. Our algorithm shares many of the data representation and clustering properties of other techniques such as Locality Preserving Projections, Laplacian Eigenmaps, and spectral clustering, yet our algorithm is much more computationally efficient. Experimental results show the efficiency and effectiveness of our algorithm.

The problem of isometry-invariant representation and comparison of surfaces is of cardinal importance in pattern recognition applications dealing with deformable objects. Particularly, in three-dimensional face recognition treating facial expressions as isometries of the facial surface allows to perform robust recognition insensitive to expressions.Isometry-invariant representation of surfaces can be constructed by isometrically embedding them into some convenient space, and carrying out the comparison in that space. Presented here is a discussion on isometric embedding into \(\mathbb{S}^{\rm 3}\), which appears to be superior over the previously used Euclidean space in sense of the representation accuracy.KeywordsFacial ExpressionFace RecognitionCanonical FormGeodesic DistanceIsometric EmbeddingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Representation learning on temporal graphs has drawn considerable research attention owing to its fundamental importance in a wide spectrum of real-world applications. Though a number of studies succeed in obtaining time-dependent representations, it still faces significant challenges. On the one hand, most of the existing methods restrict the embedding space with a certain curvature. However, the underlying geometry in fact shifts among the positive curvature hyperspherical, zero curvature Euclidean and negative curvature hyperbolic spaces in the evolvement over time. On the other hand, these methods usually require abundant labels to learn temporal representations, and thereby notably limit their wide use in the unlabeled graphs of the real applications. To bridge this gap, we make the first attempt to study the problem of self-supervised temporal graph representation learning in the general Riemannian space, supporting the time-varying curvature to shift among hyperspherical, Euclidean and hyperbolic spaces. In this paper, we present a novel self-supervised Riemannian graph neural network (SelfRGNN). Specifically, we design a curvature-varying Riemannian GNN with a theoretically grounded time encoding, and formulate a functional curvature over time to model the evolvement shifting among the positive, zero and negative curvature spaces. To enable the self-supervised learning, we propose a novel reweighting self-contrastive approach, exploring the Riemannian space itself without augmentation, and propose an edge-based self-supervised curvature learning with the Ricci curvature. Extensive experiments show the superiority of SelfRGNN, and moreover, the case study shows the time-varying curvature of temporal graph in reality.

We give a contribution to the representation problem of free-form curves and surfaces. Our proposal is an operational or kinematic approach based on the Lie group SE(3). While in Euclidean space the modelling of shape as an orbit of a point under the action of SE(3) is limited, we are embedding our problem into the conformal geometric algebra ℝ4,1 of the Euclidean space ℝ3. This embedding results in a number of advantages which makes the proposed method a universal and flexible one with respect to applications. It makes possible the robust and fast estimation of the pose of 3D objects from incomplete and noisy image data. Especially advantagous is the equivalence of the proposed shape model to that of the Fourier representations.

Concerning the learning problems of recurrent neural networks (RNNs), this paper deals with the problem of approximating a dynamical system (DS) by an RNN as one extension of the problem of approximating trajectories by an RNN. In particular, we systematically investigate how an RNN can produce a DS on the visible state space to approximate a target DS. First, it is proved that RNNs without hidden units uniquely produce a certain class of DSs. Next, a neural dynamical system (NDS) is proposed as such a DS that an RNN with hidden units can produce on the visible state space, and affine neural dynamical systems (A-NDSs) are constructed as concrete examples of NDSs. Moreover, we prove that any DS on a Euclidean space can be finitely approximated by some A-NDS with any precision, and propose adopting an A-NDS as such a DS that an RNN with hidden units produces to approximate a target DS. © 2000 Scripta Technica, Syst Comp Jpn, 31(4): 77–86, 2000

Given a hypergraph H with m hyperedges and a set X of m pins, i.e. globally fixed subspaces in Euclidean space $$\mathbb {R}^{d}$$ , a pinned subspace-incidence system is the pair (H, X), with the constraint that each pin in X lies on the subspace spanned by the point realizations in $$\mathbb {R}^d$$ of vertices of the corresponding hyperedge of H. Pinned subspace-incidence systems arise in modeling dictionary learning problems as well as biomaterials such as cell wall microfibrils. We are interested in combinatorial characterization of pinned subspace-incidence systems that are minimally rigid, i.e. those systems that are guaranteed to generically yield a locally unique realization. As is customary, this is accompanied by a characterization of generic independence as well as rigidity. Previously, such a combinatorial rigidity characterization is only known for a more restricted version of pinned subpsace-incidence systems, with H being a uniform hypergraph and pins in X being 1-dimension subspaces. In this paper, we extend the combinatorial characterization to general pinned subspace-incidence systems, with H being a non-uniform hypergraph and pins in X being subspaces with arbitrary dimension. As there are generally many data points per subspace in a dictionary learning problem, which can only be modeled with pins of dimension larger than 1, such an extension enables application to a much larger class of dictionary learning problems.

Graphs are a major tool for modeling objects with complex data structures. Devising learning algorithms that are able to handle graph representations is thus a core issue in knowledge discovery with complex data. While a significant amount of recent research has been devoted to inducing functions on the vertices of the graph, we concentrate on the task of inducing a function on the set of graphs. Application areas of such learning algorithms range from computer vision to biology and beyond. Here, we present a number of results on extending kernel methods to complex data, in general, and graph representations, in particular. With the very good performance of kernel methods on data that can easily be embedded in a Euclidean space, kernel methods have the potential to overcome some of the major weaknesses of previous approaches to learning from complex data. In order to apply kernel methods to graph data, we propose two different kernel functions and compare them on a relational reinforcement learning problem and a molecule classification problem.

A wide range of machine learning problems, including astronomical inference about galaxy clusters, scene classification, parametric statistical inference, and predictions of public opinion, can be well-modeled as learning a function on (samples from) distributions. This project explores problems in learning such functions via kernel methods, particularly for large-scale problems. When learning from large numbers of distributions, the computation of typical methods scales between quadratically and cubically, and so they are not amenable to large datasets. We investigate the approach of approximate embeddings into Euclidean spaces such that inner products in the embedding space approximate kernel values between the source distributions. We first improve the understanding of the workhorse methods of random Fourier features: we show that of the two approaches in common usage, one is strictly superior. We then present a new embedding for a class of information-theoretic distribution distances, and evaluate it and existing embeddings on several real-world applications.

We consider a class of nonsmooth optimization problems over the Stiefel manifold, in which the objective function is weakly convex in the ambient Euclidean space. Such problems are ubiquitous in engineering applications but still largely unexplored. We present a family of Riemannian subgradient-type methods---namely Riemannian subgradient, incremental subgradient, and stochastic subgradient methods---to solve these problems and show that they all have an iteration complexity of $\mathcal{O}(\varepsilon^{-4})$ for driving a natural stationarity measure below $\varepsilon$. In addition, we establish the local linear convergence of the Riemannian subgradient and incremental subgradient methods when the problem at hand further satisfies a sharpness property and the algorithms are properly initialized and use geometrically diminishing stepsizes. To the best of our knowledge, these are the first convergence guarantees for using Riemannian subgradient-type methods to optimize a class of nonconvex nonsmooth functions over the Stiefel manifold. The fundamental ingredient in the proof of the aforementioned convergence results is a new Riemannian subgradient inequality for restrictions of weakly convex functions on the Stiefel manifold, which could be of independent interest. We also show that our convergence results can be extended to handle a class of compact embedded submanifolds of the Euclidean space. Finally, we discuss the sharpness properties of various formulations of the robust subspace recovery and orthogonal dictionary learning problems and demonstrate the convergence performance of the algorithms on both problems via numerical simulations.

In Reinforcement Learning, an intelligent agent has to make a sequence of decisions to accomplish a goal. If this sequence is long, then the agent has to plan over a long horizon. While learning the optimal policy and its value function is a well studied problem in Reinforcement Learning, this paper focuses on the structure of the optimal value function and how hard it is to represent the optimal value function. We show that the generalized Rademacher complexity of the hypothesis space of all optimal value functions is dependent on the planning horizon and independent of the state and action space size. Further, we present bounds on the action-gaps of action value functions and show that they can collapse if a long planning horizon is used. The theoretical results are verified empirically on randomly generated MDPs and on a grid-world fruit collection task using deep value function approximation. Our theoretical results highlight a connection between value function approximation and the Options framework and suggest that value functions should be decomposed along bottlenecks of the MDP's transition dynamics.

This paper describes an algorithm, called CQ-learning, which learns to adapt the state representation for multi-agent systems in order to coordinate with other agents. We propose a multi-level approach which builds a progressively more advanced representation of the learning problem. The idea is that agents start with a minimal single agent state space representation, which is expanded only when necessary. In cases where agents detect conflicts, they automatically expand their state to explicitly take into account the other agents. These conflict situations are then analyzed in an attempt to find an abstract representation which generalises over the problem states. Our system allows agents to learn effective policies, while avoiding the exponential state space growth typical in multi-agent environments. Furthermore, the method we introduce to generalise over conflict states allows knowledge to be transferred to unseen and possibly more complex situations. Our research departs from previous efforts in this area of multi-agent learning because our agents combine state space generalisation with an agent-centric point of view. The algorithms that we introduce can be used in robotic systems to automatically reduce the sensor information to what is essential to solve the problem at hand. This is a must when dealing with multiple agents, since learning in such environments is a cumbersome task due to the massive amount of information, much of which may be irrelevant. In our experiments we demonstrate a simulation of such environments using various gridworlds.

Corporate earnings are a crucial indicator for investment and business valuation. Despite their importance and the fact that classic econometric approaches fail to match analyst forecasts by orders of magnitude, the automatic prediction of corporate earnings from public data is not in the focus of current machine learning research. In this paper, we present for the first time a fully automatized machine learning method for earnings prediction that at the same time a) only relies on publicly available data and b) can outperform human analysts. The latter is shown empirically in an experiment involving all S&P 100 companies in a test period from 2008 to 2012. The approach employs a simple linear regression model based on a novel feature space of stock market prices and their pairwise correlations. With this work we follow the recent trend of nowcasting, i.e., of creating accurate contemporary forecasts of undisclosed target values based on publicly observable proxy variables.

Breast cancer is one of the most frequently occurring female cancer types and represents a major cause of death among women worldwide. Breast cancer is heterogeneous in both molecular characteristics and clinical outcomes for its different molecular subtypes. High-throughput technologies facilitated the fast accumulations of the multiple Omic data for cancer patients. These data sources posed a computational challenge for the efficient integrated multi-Omic analysis. The existing studies usually investigated the differential representation or machine learning problems using a single type of Omic data. This study hypothesized that different Omic types contributed complementary information to each other, and their integrated analysis may improve the single-Omic models. An efficient logistic regression-based multi-Omic integrated analysis method (ELMO) was proposed to integrate the RNA-seq and DNA methylation data to detect the breast cancer intrinsic subtypes. ELMO achieved the highest accuracy with a smaller number of features compared with the existing filter and wrapper feature selection methods in this study. The experimental data supported our hypothesis that multi-Omic models outperformed the single-Omic ones.

In this paper, we formulate particle filter-based object tracking as an exclusive sparse learning problem that exploits contextual information. To achieve this goal, we propose the context-aware exclusive sparse tracker (CEST) to model particle appearances as linear combinations of dictionary templates that are updated dynamically. Learning the representation of each particle is formulated as an exclusive sparse representation problem, where the overall dictionary is composed of multiple group dictionaries that can contain contextual information. With context, CEST is less prone to tracker drift. Interestingly, we show that the popular L1 tracker is a special case of our CEST formulation. The proposed learning problem is efficiently solved using an accelerated proximal gradient method that yields a sequence of closed form updates. To make the tracker much faster, we reduce the number of learning problems to be solved by using the dual problem to quickly and systematically rank and prune particles in each frame. We test our CEST tracker on challenging benchmark sequences that involve heavy occlusion, drastic illumination changes, and large pose variations. Experimental results show that CEST consistently outperforms state-of-the-art trackers.

We present a subspace learning method, called local discriminant embedding with tensor representation (LDET), that addresses simultaneously the generalization and data representation problems in subspace learning. LDET learns multiple interrelated subspaces for obtaining a lower-dimensional embedding by incorporating both class label information and neighborhood information. By encoding each object as a second- or higher-order tensor, LDET can capture higher-order structures in the data without requiring a large sample size. Extensive empirical studies have been performed to compare LDET with a second- or third-order tensor representation and the original LDE on their face recognition performance. Not only does LDET have a lower computational complexity than LDE, but LDET is also superior to LDE in terms of its recognition accuracy.